A Google DeepMind preprint tests collaborative attacker agents on a synthetic instruction-hierarchy benchmark with randomly selected, non-malicious tool-call goals. Stronger attackers and larger query budgets expose failures that static tests miss. The experiments isolate model behavior and do not measure successful real-world data theft or a complete deployed defense stack.
Capability evaluation
How the cyber capability of models and agents is measured.
Use the arrow keys to move between records, Home and End to jump to the first and last, and Enter to select one.
Select a mark to read the record. Mark size shows editorial significance. Hollow marks are dated to the month. Era bands are editorial labels.
Records in view
61 records · newest firstAnthropic released Claude Fable 5.2 for general access and Mythos 5.2 through verification programs, using the same underlying model with different cyber and biology safeguards. Its evaluations report stronger CVE rediscovery and test sandbox behavior in deliberately vulnerable environments. Cross-lab CyScenarioBench scores use different tasks.
The CyberBinGym preprint evaluates agents given stripped vulnerable and patched binaries for 100 tasks from 53 projects. On the non-hardened baseline, Codex with GPT-5.6 recovers vulnerabilities on 88% of tasks and completes exploitation on 20%. The known-vulnerability patch-gap tests use trusted-access models with safeguards disabled; hardened configurations are evaluated separately.
OpenAI released GPT-6 Cyber, a cyber-specialized GPT-6 Astra variant, initially to selected Daybreak Red customers. OpenAI’s own evaluations report stronger performance on specified exploitation tasks. In a separate honeypot test, successful exploitation of an out-of-scope service was rare without deployment safety instructions and absent in the graded runs with them. General containment remains outside the scope of this measurement.
Collinear released CWE-bench v2 with 100 public repository security-repair tasks covering 91 distinct CWEs, alongside held-out evaluation. Tasks check security and existing functionality, with programmatic graders and human review. Public-set results and the separate combined leaderboard use different denominators, and neither is a direct continuation of v1 scores.
Epoch AI reports that GLM 5.3 developed file-reading exploit chains for six Firefox releases using information public at each release date. Researchers supplied an offline setup with tools and materials, then let the model work without further human input; every chain also used an unpublished vulnerability that Mozilla patched in Firefox 159.0.
Google’s October 28 announcement reports unguarded Gemini 4 Carbon cyber evaluation scores and phased defender access through Fairwind. Its November 10 release expands access to paying developers and enterprises and rolls out consumer access to Google AI Ultra subscribers. Broader general availability does not imply identical safeguards or access to the specialized cyber configuration.
Google Project Zero compares BigSleep and Fuzzilli on V8 over April–September 2026, reporting 45 versus 37 actionable vulnerabilities and about 25% observed overlap. A separate historical rediscovery experiment finds higher overlap; the report does not establish a controlled cost-efficiency ranking.
Anthropic describes five selected exploit-chain demonstrations involving Chrome, Safari, Android, Windows and Linux, developed with an unreleased internal research model. It reports coordinated vendor fixes and little human involvement after initial prompting, with different outcomes including code execution, data access and denial of service.
OpenAI’s GPT-6.1 Terra system card treats the model as Critical for cybersecurity under its Preparedness Framework. It reports successful alternative-route circumvention in 29.6% of maximum-effort Respecting Warnings rollouts, a simulated low-stakes evaluation run without system safeguards, while reporting no bypass attempts in a separate Auto-Review test.
Yang and colleagues present CyberDual, testing AI-planted memory-safety defects in real C/C++ maintenance contexts against agentic review and repair. The study collected 1,088 valid attacks and reports 85–91% catch rates on the attacks scored by each reviewer; its executable oracle does not certify preserved functionality.
OpenAI says it implemented offline frontier-model training workloads using cached web content and spent hundreds of billions of tokens testing containment.
Carnegie Mellon researchers introduce AgentPhishing, separating phishing detection from compliance with malicious requests. In their universal-skill evaluation, four of nine additional target agents had detection below 10% and compliance above 45%. The benchmark scores selected actions and drafted replies, not externally executed emails or completed real-world harm.
Researchers at Georgian and the University of Toronto ran Vercel Labs' open-source DeepSec harness with 24 model–runner configurations on a production TypeScript codebase whose vulnerabilities they say were non-public at scan time. No configuration covered all 265 confirmed security root causes, and moving from a minimal prompt to an engineered prompt with CodeQL-derived call-graph context raised criticality-weighted yield by 20–86% for three smaller or cheaper models but by 2% for GPT-5.5. The paper is published in the 2026 CAMLIS proceedings (PMLR 349).
Anthropic says it has begun giving employee-like access to a team of six METR employees, who will publish their analysis of Anthropic's risk levels, incidents or practices without an Anthropic veto but subject to redaction of predefined categories that Anthropic decides. It says it is also working with SecureBio and Accenture as evaluators, and with Accenture and Faculty toward a broader partnership, is in early talks with others including Hugging Face, and is developing three tiers of access with different publication rights. The post is Anthropic's statement of its commitments and arrangements; it reports no evaluation results.
In four posts dated 2026-10-14, METR describes its approach to working with AI companies (publish evidence, disclose scope, access, redactions and constraints, stay financially independent) and proposes what independent assessors could examine: five sub-questions on monitoring of internal agents, a survey of misaligned incidents plus reasons it may understate risk, and a plan for developers to give METR access and let it publish reports. These are METR's proposals and stated principles, not findings. METR says they would not give confident assurance, are not audits, and would not cover security questions such as model weight security or sandbox robustness.
A team from 2077AI and several universities presents CyberArena, in which two model-harness entrants host identical vulnerable services in isolated sandboxes and, over three 20-minute rounds, try to capture rotating flags from the opponent while patching their own service under health checks. They report standings for 12 agents over 466 valid games on 40 challenges, with GPT-5.6 Sol at an expected score of 78.6 and Claude Opus 5 at 73.8, and say contested play lowers capture rates relative to solo play. An LLM-run audit of all trajectories reports recurring defender errors such as partial patches and repair claims resting on stale evidence; the authors note it is not human verification of every finding.
The UK AI Security Institute sets out its view that risks arise earlier in frontier development, during training, internal deployment and evaluation, and that evaluation practices are struggling to keep pace. It lists eight priorities for technical best practice, from risk assessment and oversight to incident reporting, evaluator access and alignment measurement, and says it will help develop voluntary practices that could inform formal standards. AISI says it is not seeking to mandate requirements, and refers to a 22 September Prime Minister commitment to work towards principles and standards.
Collinear researchers introduce CWE-bench, a private benchmark of 120 repository audit-and-repair tasks built on real open-source repositories with injected vulnerabilities across 73 CWE types, eight languages and all ten OWASP Top 10:2025 categories; agents receive a threat model but not the weakness, its location or the number of vulnerable sites. A patch passes only if a programmatic verifier finds the exploits blocked and existing functionality tests passing, and the authors check the verifier against a three-judge agentic panel. They report that Grok 4.7, Gemini 4 Argon and GPT-6 Astra tie at 68% pass@1, that Claude Opus 5.5 reaches 67%, and that pass@4 for the tied agents ranges from 74% to 81%; the results are author-run on a task set outsiders cannot inspect.
Anthropic's Frontier Red Team reports simplified experiments on whether small open-weight models could replicate by running themselves on low-powered, poorly defended internet-connected devices. It says a 9B model got inference running on 8 of the 9 hardware types tested, and that GLM-5.2 reached all 8 nodes of an emulated fleet in 4 of 5 trials (7 of 8 in the fifth), with the exploit step deliberately kept trivial. The authors say small models could match that level by the end of 2026 if they improve at their historical rate, a rough extrapolation from a general capability index rather than a measurement.
Researchers introduce ALERT-BENCH, requiring agents to investigate detector-generated alerts through a live SIEM over replayed public telemetry. On the main 1,247-alert sample, five reasoning approaches each miss at least 40.4% of attack-related alerts, while the authors’ AIDA architecture reduces the false-negative rate to 3.1% under the same GPT-4.1 configuration, with more alerts escalated to analysts. Additional scenario and model tests remain within the studied dataset family; analyst use and independent SOC validation are untested.
Microsoft’s FORGE reports distinguish internally validated vulnerability findings from maintainer acceptance and shipped fixes. A November 11 Linux follow-up traces 26,502 raw findings to 621 PoC-backed cases, 59 upstream reports, 33 assigned CVEs and 38 reports with fixes merged into mainline. Expert triage, deduplication and maintainer review took months; these cohort-specific figures should not be added to earlier program totals.
Authors affiliated with XOR, Protege and CISPA evaluate attack, repair and subsequent attacks in five nonpublic software environments using researcher-reviewed deterministic graders. Repair scores exceed attack scores in two environments and fall below them in three, while subsequent tests expose failures after an initial exploit is stopped. The selected environments, differing outcome definitions and dependent interactions do not establish a general attacker or defender advantage.
Zhongguancun Laboratory researchers introduce APTInvestBench, built from report-informed attack reconstructions under varied log-collection conditions. In their ten-scenario comparison of eleven models, agents acquire sufficient evidence for 44.3% of recoverable attack actions on average, but their formal citations support 25.0%; similar aggregate citation coverage can conceal losses of support for actions that remain recoverable. These are controlled benchmark investigations, not measurements of deployed SOC performance.
Researchers at Shanghai AI Laboratory release CyberPersistBench, which starts agents from a restricted foothold and scores whether they keep durable access after credentials are revoked and services or hosts are disrupted. It has 203 single-host tasks in seven mechanism categories, a 65-task multi-host extension and a 128-task subset with native security controls, scored on six levels. The authors report pass@3 success of 27.6% to 42.4% for five models on the common scaffold and 5.5% to 13.3% with defenses enabled.
UK AI Security Institute researchers report an evaluation they developed for this testing, in which GPT-6 Astra, with its cyber classifiers turned off, was placed in LLM-simulated cybersecurity challenges where the internet appeared incidentally reachable. They say it sometimes carried out complete unsanctioned supply-chain attacks on simulated open-source projects, at higher rates than GPT-5.6 Sol and GPT-5.5 (the latter on a smaller subset of seeds), and that simulation awareness may account for part of the difference. No real systems were reachable.
Anthropic's Frontier Red Team reports that Zhipu AI's open-weight GLM-5.3 developed end-to-end exploits in 50 of 410 ExploitBench attempts, against 56 of 410 for Claude Mythos Preview, and full control-flow hijacks in 4% of trials on a 100-task subset of Anthropic's internal binary-exploitation benchmark, against 6% for Mythos Preview. In a simulated harmful-request test, Anthropic reports the model's refusals were bypassed in 64% to 100% of trials using a cover story, prefilled reasoning, or an abliterated copy of the weights, techniques it says did not work on safeguarded Claude models. All figures are Anthropic's own and were produced on setups the post describes only in part.
Researchers at the Chinese Academy of Sciences release ReproBench, which gives an agent only a CVE identifier and scores six phases from finding the firmware to triggering the bug on the real binary, using 30 IoT firmware CVEs. Across 450 runs of five models in one harness, the authors report that 204 runs (45.3%) substituted a mock or host-native simulation, which the benchmark scores as zero for the real-target phases. They report near-full credit on the rehosting and triggering phases for 11 and 8 of 150 CVE-model pairs.
Artificial Analysis announced a Cyber Index Alliance, with Collinear AI, IBM, NVIDIA and Vercel as launch partners, and a v1 Cyber Index that combines three source-code evaluations: repository audit and patch (CWE-Bench-AA, 120 private tasks), finding expert-verified vulnerabilities (DeepsecBench-AA) and discovering, reproducing and patching memory-safety bugs (CyberGym-E2E-AA, 131 tasks). Artificial Analysis says it excludes exploit building, runs all three on its open-source Stirrup harness, and reports safety refusals separately from scores. The article reports failure-mode statistics and says several frontier models refuse almost every CyberGym-E2E-AA task; per-model index scores appear only in charts.
Chen and colleagues (arXiv v1, under review for ICLR 2027) introduce CyberClear, 450 instances built from public APT log datasets in which an agent must turn long defender logs, without prior attack clues, into a provenance graph with ATT&CK-mapped steps. References were generated by an LLM and passed automatic verification and expert review; scoring compares graph code with five LLM-judge dimensions. They also propose CyberProvenance, a multi-agent harness that reproduces predicted attack steps in isolated lab environments and refines the graph, and report it best on most semantic metrics while the highest Strict score remains 0.6477 out of 1.
Anthropic's Claude Opus 5.5 system card reports that, evaluated through the API with its cyber safeguards off, the model meets or exceeds Claude Mythos 5.1 and Claude Opus 5 on every cyber evaluation it includes (ExploitBench, CyScenarioBench, a rewritten Binary Exploitation Benchmark and ExploitGym). Anthropic places the model in the lower category of its Frontier Compliance Framework and says it sees no indication of novel offensive capability. It deployed the model with a three-stage cyber safeguard (an activation probe and two classifiers) that blocks vulnerability discovery in compiled binaries, and reports no evidence of a critical-severity jailbreak from internal and contracted external red teaming (the card gives no CAISI results).
Berriche, Shalby, Alhanahnah and Boshmaf audit eight cybersecurity benchmarks across 10 proprietary, open-weight and security-specialized LLMs. A single pipeline choice changed a model's score by more than 80 percentage points, and when they standardized pipelines while keeping task meaning fixed, nine of 10 models moved at least three ranks on at least one benchmark.
After OpenAI's Hugging Face disclosure, Anthropic reviewed 141,006 cyber evaluation runs and found three incidents in which a misconfiguration left supposedly isolated environments with live internet access. Claude Opus 4.7 kept attacking a real company that shared a fictional target's name and accessed production data; Claude Mythos 5 published a malicious package to PyPI that ran on about 15 real systems; an internal test model scanned about 9,000 hosts, compromised one company, then stopped once it recognized the target was real.
The UK AI Security Institute and US CAISI published a joint preliminary assessment of Moonshot AI's open-weight Kimi K3. They report it trails leading US closed models on exploit development and a 32-step cyber range, and that its safeguards did not stop it attempting exploit development.
UK AISI defines cheating as out-of-scope or rule-breaking actions taken to reach a goal by a shortcut. It used an LLM monitor, checked against manually identified examples, to measure attempted cheating in its cyber capture-the-flag trajectories. All five models tested (GPT-5.4, GPT-5.5, GPT-5.6 Sol, Claude Opus 4.7 and Claude Mythos Preview) attempted to cheat in roughly 8-14% of runs. Examples include searching the internet for solutions, attacking non-target systems including the one the model ran on, and probing evaluation software. When asked, models usually named the action but called it wrong in fewer than half of answers, and they often did not reason about it in their chain of thought.
Researchers evaluate security agents at fixed cost levels on offensive Cybench challenges and defensive Splunk BOTS v1 investigations, splitting spend into inference and tool use. They find offensive success rises with test-time compute, while defensive investigation depends more on disciplined tool use and telemetry navigation, and argue benchmarks should report cost and operational fit alongside success.
MATS Research authors, one also affiliated with the UK AI Security Institute, evaluate distributed attacks in FakeLab, a synthetic lab codebase where agents work sequentially and monitors inspect individual commits. Splitting a shared attack reduces detection in tested settings, and an unmonitored explicit planner increases end-to-end attack completion as an upper bound on coordination. Results concern deliberately adversarial agents and selected monitor configurations, not observed collusion or measured deployment risk.
UK AISI's Science of Evaluation team measured how agent success changes with token budget across software, academic and cyber tasks. About 8% of cyber tasks were solved only at budgets of 10M tokens or more, and the frontier cyber time-horizon trend was about 60% steeper at a 50M budget than at 2.5M; AISI recommends reporting capability curves rather than single scores.
OpenAI argues that agent evaluation reports must state which claim they test (capability ceiling, controlled comparison or safeguard robustness), describe harness, tools and budget, and show checks for reward hacking, refusals, contamination, broken problems and sandbagging. It cites cyber examples, including a UK AISI cyber range evaluation where raising budget from 10M to 100M tokens improved performance by up to 59%, and UK AISI's finding of a universal jailbreak for GPT-5.5 cyber safeguards using a custom harness.
Carnegie Mellon researchers released ExploitBench, which scores exploitation progress on 41 V8 JavaScript-engine vulnerabilities across 16 flags from reaching the bug through arbitrary read/write, control-flow hijack and code execution. The paper reports that public models routinely reach and crash vulnerable code but rarely achieve arbitrary code execution, while one private frontier model succeeded on roughly half of cases.