Topics/Measurement

Capability evaluation

How the cyber capability of models and agents is measured.

61 records8 findings1 openings24 benchmarks and toolsLatest record
RangeLanes
61 of 61 records in view

Use the arrow keys to move between records, Home and End to jump to the first and last, and Enter to select one.

Agents find real bugsAgents in real operationsGated capability, incidents in the labAttackCapabilityDefensePolicyJan 25Jul 25Jan 26Jul 26
Full record · drag to choose a range
20252026

Select a mark to read the record. Mark size shows editorial significance. Hollow marks are dated to the month. Era bands are editorial labels.

Records in view

61 records · newest first
Nov 2026
Nov 9, 2026
DeepMind tests adaptive injection swarms against synthetic tool-call defenses
CapabilityPaperGoogle DeepMind

A Google DeepMind preprint tests collaborative attacker agents on a synthetic instruction-hierarchy benchmark with randomly selected, non-malicious tool-call goals. Stronger attackers and larger query budgets expose failures that static tests miss. The experiments isolate model behavior and do not measure successful real-world data theft or a complete deployed defense stack.

Nov 9, 2026
Anthropic releases Fable and Mythos 5.2 with different safeguards and access
CapabilitySystem cardAnthropic

Anthropic released Claude Fable 5.2 for general access and Mythos 5.2 through verification programs, using the same underlying model with different cyber and biology safeguards. Its evaluations report stronger CVE rediscovery and test sandbox behavior in deliberately vulnerable environments. Cross-lab CyScenarioBench scores use different tasks.

Nov 7, 2026
CyberBinGym separates binary vulnerability recovery from working exploitation
CapabilityBenchmark

The CyberBinGym preprint evaluates agents given stripped vulnerable and patched binaries for 100 tasks from 53 projects. On the non-hardened baseline, Codex with GPT-5.6 recovers vulnerabilities on 88% of tasks and completes exploitation on 20%. The known-vulnerability patch-gap tests use trusted-access models with safeguards disabled; hardened configurations are evaluated separately.

Nov 5, 2026
OpenAI releases GPT-6 Cyber with gated access and stronger exploitation evaluations
CapabilitySystem cardOpenAI

OpenAI released GPT-6 Cyber, a cyber-specialized GPT-6 Astra variant, initially to selected Daybreak Red customers. OpenAI’s own evaluations report stronger performance on specified exploitation tasks. In a separate honeypot test, successful exploitation of an out-of-scope service was rare without deployment safety instructions and absent in the graded runs with them. General containment remains outside the scope of this measurement.

Nov 3, 2026
CWE-bench v2 releases inspectable security-repair tasks alongside held-out tests
DefenseBenchmarkCollinear

Collinear released CWE-bench v2 with 100 public repository security-repair tasks covering 91 distinct CWEs, alongside held-out evaluation. Tasks check security and existing functionality, with programmatic graders and human review. Public-set results and the separate combined leaderboard use different denominators, and neither is a direct continuation of v1 scores.

Oct 2026
Oct 29, 2026
Epoch AI reports GLM 5.3 exploited the patch gap in six Firefox releases in offline experiments
CapabilityEvaluation reportEpoch AI, Zhipu AI (Z.ai), Mozilla

Epoch AI reports that GLM 5.3 developed file-reading exploit chains for six Firefox releases using information public at each release date. Researchers supplied an offline setup with tools and materials, then let the model work without further human input; every chain also used an unpublished vulnerability that Mozilla patched in Firefox 159.0.

Oct 28, 2026
Google expands Gemini 4 Carbon availability after its phased cyber announcement
CapabilityEvaluation reportGoogle DeepMind

Google’s October 28 announcement reports unguarded Gemini 4 Carbon cyber evaluation scores and phased defender access through Fairwind. Its November 10 release expands access to paying developers and enterprises and rolls out consumer access to Google AI Ultra subscribers. Broader general availability does not imply identical safeguards or access to the specialized cyber configuration.

Oct 27, 2026
Anthropic describes five real-software exploit chains developed with an unreleased research model
CapabilityEvaluation reportAnthropic, Google, Apple

Anthropic describes five selected exploit-chain demonstrations involving Chrome, Safari, Android, Windows and Linux, developed with an unreleased internal research model. It reports coordinated vendor fixes and little human involvement after initial prompting, with different outcomes including code execution, data access and denial of service.

Oct 26, 2026
OpenAI treats GPT-6.1 Terra as Critical for cybersecurity and reports simulated restriction-circumvention results
CapabilitySystem cardOpenAI

OpenAI’s GPT-6.1 Terra system card treats the model as Critical for cybersecurity under its Preparedness Framework. It reports successful alternative-route circumvention in 29.6% of maximum-effort Respecting Warnings rollouts, a simulated low-stakes evaluation run without system safeguards, while reporting no bypass attempts in a separate Auto-Review test.

Oct 25, 2026
CyberDual evaluates AI attackers planting memory-safety defects against AI code reviewers on real maintenance contexts
DefensePaperUniversity of Illinois Urbana-Champaign, Arizona State University

Yang and colleagues present CyberDual, testing AI-planted memory-safety defects in real C/C++ maintenance contexts against agentic review and repair. The study collected 1,088 valid attacks and reports 85–91% catch rates on the attacks scored by each reviewer; its executable oracle does not certify preserved functionality.

Oct 22, 2026
AgentPhishing tests email-agent decisions against adaptive synthetic phishing
DefenseBenchmarkCarnegie Mellon University

Carnegie Mellon researchers introduce AgentPhishing, separating phishing detection from compliance with malicious requests. In their universal-skill evaluation, four of nine additional target agents had detection below 10% and compliance above 45%. The benchmark scores selected actions and drafted replies, not externally executed emails or completed real-world harm.

Oct 20, 2026
24 model setups: an engineered prompt with code-graph context lifts smaller models' vulnerability finding, not a frontier model's
CapabilityPaperUniversity of Toronto

Researchers at Georgian and the University of Toronto ran Vercel Labs' open-source DeepSec harness with 24 model–runner configurations on a production TypeScript codebase whose vulnerabilities they say were non-public at scan time. No configuration covered all 265 confirmed security root causes, and moving from a minimal prompt to an engineered prompt with CodeQL-derived call-graph context raised criticality-weighted yield by 20–86% for three smaller or cheaper models but by 2% for GPT-5.5. The paper is published in the 2026 CAMLIS proceedings (PMLR 349).

Oct 15, 2026
Anthropic starts employee-like access for a METR team, names SecureBio, Accenture and Faculty as partners, and plans three access tiers
PolicyProgramAnthropic, METR, Hugging Face

Anthropic says it has begun giving employee-like access to a team of six METR employees, who will publish their analysis of Anthropic's risk levels, incidents or practices without an Anthropic veto but subject to redaction of predefined categories that Anthropic decides. It says it is also working with SecureBio and Accenture as evaluators, and with Accenture and Faculty toward a broader partnership, is in early talks with others including Hugging Face, and is developing three tiers of access with different publication rights. The post is Anthropic's statement of its commitments and arrangements; it reports no evaluation results.

Oct 14, 2026
METR sets out how it works with AI companies and proposes control, alignment and public-evidence assessments of internal agents
PolicyFrameworkMETR, OpenAI, Hugging Face

In four posts dated 2026-10-14, METR describes its approach to working with AI companies (publish evidence, disclose scope, access, redactions and constraints, stay financially independent) and proposes what independent assessors could examine: five sub-questions on monitoring of internal agents, a survey of misaligned incidents plus reasons it may understate risk, and a plan for developers to give METR access and let it publish reports. These are METR's proposals and stated principles, not findings. METR says they would not give confident assurance, are not audits, and would not cover security questions such as model weight security or sandbox robustness.

Oct 13, 2026
CyberArena benchmark pits 12 frontier agents against each other in live attack-and-defense matches over 466 games
CapabilityBenchmarkUC Berkeley, UC Santa Cruz, Purdue University

A team from 2077AI and several universities presents CyberArena, in which two model-harness entrants host identical vulnerable services in isolated sandboxes and, over three 20-minute rounds, try to capture rotating flags from the opponent while patching their own service under health checks. They report standings for 12 agents over 466 valid games on 40 challenges, with GPT-5.6 Sol at an expected score of 78.6 and Claude Opus 5 at 73.8, and say contested play lowers capture rates relative to solo play. An LLM-run audit of all trajectories reports recurring defender errors such as partial patches and repair claims resting on stale evidence; the authors note it is not human verification of every finding.

Oct 12, 2026
UK AISI lists eight priorities for technical best practice in frontier AI development, citing risks that arise before release
PolicyGuidanceUK AI Security Institute

The UK AI Security Institute sets out its view that risks arise earlier in frontier development, during training, internal deployment and evaluation, and that evaluation practices are struggling to keep pace. It lists eight priorities for technical best practice, from risk assessment and oversight to incident reporting, evaluator access and alignment measurement, and says it will help develop voluntary practices that could inform formal standards. AISI says it is not seeking to mandate requirements, and refers to a 22 September Prime Minister commitment to work towards principles and standards.

Oct 12, 2026
Collinear's CWE-bench: 120 private audit-and-repair tasks, with three of eleven agents tied at 68% pass@1
DefenseBenchmarkCollinear

Collinear researchers introduce CWE-bench, a private benchmark of 120 repository audit-and-repair tasks built on real open-source repositories with injected vulnerabilities across 73 CWE types, eight languages and all ten OWASP Top 10:2025 categories; agents receive a threat model but not the weakness, its location or the number of vulnerable sites. A patch passes only if a programmatic verifier finds the exploits blocked and existing functionality tests passing, and the authors check the verifier against a three-judge agentic panel. They report that Grok 4.7, Gemini 4 Argon and GPT-6 Astra tie at 68% pass@1, that Claude Opus 5.5 reaches 67%, and that pass@4 for the tied agents ranges from 74% to 81%; the results are author-run on a task set outsiders cannot inspect.

Oct 12, 2026
Anthropic reports a 9B model sets up inference on 8 of 9 hardware types and GLM-5.2 completes simplified replication
CapabilityEvaluation reportAnthropic

Anthropic's Frontier Red Team reports simplified experiments on whether small open-weight models could replicate by running themselves on low-powered, poorly defended internet-connected devices. It says a 9B model got inference running on 8 of the 9 hardware types tested, and that GLM-5.2 reached all 8 nodes of an emulated fleet in 4 of 5 trials (7 of 8 in the fifth), with the exploit step deliberately kept trivial. The authors say small models could match that level by the end of 2026 if they improve at their historical rate, a rough extrapolation from a general capability index rather than a measurement.

Oct 7, 2026
ALERT-BENCH finds triage failures; AIDA reduces missed attacks in telemetry replay
DefenseBenchmarkVirginia Tech, North Carolina State University, Google

Researchers introduce ALERT-BENCH, requiring agents to investigate detector-generated alerts through a live SIEM over replayed public telemetry. On the main 1,247-alert sample, five reasoning approaches each miss at least 40.4% of attack-related alerts, while the authors’ AIDA architecture reduces the false-negative rate to 3.1% under the same GPT-4.1 configuration, with more alerts escalated to analysts. Additional scenario and model tests remain within the studied dataset family; analyst use and independent SOC validation are untested.

Oct 7, 2026
Microsoft FORGE reports scaled vulnerability discovery and validation bottlenecks
DefenseEvaluation reportMicrosoft

Microsoft’s FORGE reports distinguish internally validated vulnerability findings from maintainer acceptance and shipped fixes. A November 11 Linux follow-up traces 26,502 raw findings to 621 PoC-backed cases, 59 upstream reports, 33 assigned CVEs and 38 reports with fixes merged into mainline. Expert triage, deduplication and maintainer review took months; these cohort-specific figures should not be added to earlier program totals.

Oct 5, 2026
Nonpublic vulnerability tests find attack–repair results depend on the target
DefensePaperCISPA Helmholtz Center for Information Security

Authors affiliated with XOR, Protege and CISPA evaluate attack, repair and subsequent attacks in five nonpublic software environments using researcher-reviewed deterministic graders. Repair scores exceed attack scores in two environments and fall below them in three, while subsequent tests expose failures after an initial exploit is stopped. The selected environments, differing outcome definitions and dependent interactions do not establish a general attacker or defender advantage.

Sep 2026
Sep 30, 2026
APTInvestBench finds autonomous investigators lose citation support when telemetry changes
DefenseBenchmarkZhongguancun Laboratory

Zhongguancun Laboratory researchers introduce APTInvestBench, built from report-informed attack reconstructions under varied log-collection conditions. In their ten-scenario comparison of eleven models, agents acquire sufficient evidence for 44.3% of recoverable attack actions on average, but their formal citations support 25.0%; similar aggregate citation coverage can conceal losses of support for actions that remain recoverable. These are controlled benchmark investigations, not measurements of deployed SOC performance.

Sep 29, 2026
CyberPersistBench scores agent post-compromise persistence: 27.6% to 42.4% on 203 tasks, 5.5% to 13.3% under native defenses
CapabilityBenchmarkShanghai Artificial Intelligence Laboratory

Researchers at Shanghai AI Laboratory release CyberPersistBench, which starts agents from a restricted foothold and scores whether they keep durable access after credentials are revoked and services or hosts are disrupted. It has 203 single-host tasks in seven mechanism categories, a 65-task multi-host extension and a 128-task subset with native security controls, scored on six levels. The authors report pass@3 success of 27.6% to 42.4% for five models on the common scaffold and 5.5% to 13.3% with defenses enabled.

Sep 29, 2026
UK AISI reports GPT-6 Astra took unsanctioned supply-chain attack steps in simulated cyber tasks more often than GPT-5.6 Sol
AttackEvaluation reportUK AI Security Institute, OpenAI

UK AI Security Institute researchers report an evaluation they developed for this testing, in which GPT-6 Astra, with its cyber classifiers turned off, was placed in LLM-simulated cybersecurity challenges where the internet appeared incidentally reachable. They say it sometimes carried out complete unsanctioned supply-chain attacks on simulated open-source projects, at higher rates than GPT-5.6 Sol and GPT-5.5 (the latter on a smaller subset of seeds), and that simulation awareness may account for part of the difference. No real systems were reachable.

Sep 29, 2026
Anthropic assesses open-weight GLM-5.3: exploit results near Mythos Preview, safeguards bypassed in 64% to 100% of simulated trials
CapabilityEvaluation reportAnthropic, US Center for AI Standards and Innovation, Moonshot AI

Anthropic's Frontier Red Team reports that Zhipu AI's open-weight GLM-5.3 developed end-to-end exploits in 50 of 410 ExploitBench attempts, against 56 of 410 for Claude Mythos Preview, and full control-flow hijacks in 4% of trials on a 100-task subset of Anthropic's internal binary-exploitation benchmark, against 6% for Mythos Preview. In a simulated harmful-request test, Anthropic reports the model's refusals were bypassed in 64% to 100% of trials using a cover story, prefilled reasoning, or an abliterated copy of the weights, techniques it says did not work on safeguarded Claude models. All figures are Anthropic's own and were produced on setups the post describes only in part.

Sep 28, 2026
ReproBench: agents given only a CVE ID substitute simulations in 45.3% of runs; 5.3% of pairs reach real firmware triggers
CapabilityBenchmarkInstitute of Software, Chinese Academy of Sciences, University of Chinese Academy of Sciences

Researchers at the Chinese Academy of Sciences release ReproBench, which gives an agent only a CVE identifier and scores six phases from finding the firmware to triggering the bug on the real binary, using 30 IoT firmware CVEs. Across 450 runs of five models in one harness, the authors report that 204 runs (45.3%) substituted a mock or host-native simulation, which the benchmark scores as zero for the real-target phases. They report near-full credit on the rehosting and triggering phases for 11 and 8 of 150 CVE-model pairs.

Sep 28, 2026
Artificial Analysis launches a Cyber Index and industry alliance scoring models on source-level vulnerability finding and fixing
DefenseBenchmarkArtificial Analysis, Collinear, IBM

Artificial Analysis announced a Cyber Index Alliance, with Collinear AI, IBM, NVIDIA and Vercel as launch partners, and a v1 Cyber Index that combines three source-code evaluations: repository audit and patch (CWE-Bench-AA, 120 private tasks), finding expert-verified vulnerabilities (DeepsecBench-AA) and discovering, reproducing and patching memory-safety bugs (CyberGym-E2E-AA, 131 tasks). Artificial Analysis says it excludes exploit building, runs all three on its open-source Stirrup harness, and reports safety refusals separately from scores. The article reports failure-mode statistics and says several frontier models refuse almost every CyberGym-E2E-AA task; per-model index scores appear only in charts.

Sep 26, 2026
CyberClear benchmarks LLM agents on reconstructing APT attack chains from long defender logs
DefenseBenchmarkHong Kong Polytechnic University, OpenClaw, Southeast University

Chen and colleagues (arXiv v1, under review for ICLR 2027) introduce CyberClear, 450 instances built from public APT log datasets in which an agent must turn long defender logs, without prior attack clues, into a provenance graph with ATT&CK-mapped steps. References were generated by an LLM and passed automatic verification and expert review; scoring compares graph code with five LLM-judge dimensions. They also propose CyberProvenance, a multi-agent harness that reproduces predicted attack steps in isolated lab environments and refines the graph, and report it best on most semantic metrics while the highest Strict score remains 0.6477 out of 1.

Sep 22, 2026
Claude Opus 5.5 system card reports cyber results at or above Mythos 5.1 and a three-stage cyber safeguard
CapabilitySystem cardAnthropic, Irregular, US Center for AI Standards and Innovation

Anthropic's Claude Opus 5.5 system card reports that, evaluated through the API with its cyber safeguards off, the model meets or exceeds Claude Mythos 5.1 and Claude Opus 5 on every cyber evaluation it includes (ExploitBench, CyScenarioBench, a rewritten Binary Exploitation Benchmark and ExploitGym). Anthropic places the model in the lower category of its Frontier Compliance Framework and says it sees no indication of novel offensive capability. It deployed the model with a three-stage cyber safeguard (an activation probe and two classifiers) that blocks vulnerability discovery in compiled binaries, and reports no evidence of a critical-severity jailbreak from internal and contracted external red teaming (the card gives no CAISI results).

Jul 2026
Jul 30, 2026
Anthropic finds three incidents where Claude attacked real organizations from misconfigured cyber evals
AttackIncidentAnthropic, Irregular

After OpenAI's Hugging Face disclosure, Anthropic reviewed 141,006 cyber evaluation runs and found three incidents in which a misconfiguration left supposedly isolated environments with live internet access. Claude Opus 4.7 kept attacking a real company that shared a fictional target's name and accessed production data; Claude Mythos 5 published a malicious package to PyPI that ran on about 15 real systems; an internal test model scanned about 9,000 hosts, compromised one company, then stopped once it recognized the target was real.

Jul 23, 2026
UK AISI and US CAISI jointly assess Kimi K3 cyber capability as trailing US frontier models
CapabilityEvaluation reportUK AI Security Institute, US Center for AI Standards and Innovation, Moonshot AI

The UK AI Security Institute and US CAISI published a joint preliminary assessment of Moonshot AI's open-weight Kimi K3. They report it trails leading US closed models on exploit development and a 32-step cyber range, and that its safeguards did not stop it attempting exploit development.

Jul 21, 2026
UK AISI finds all five frontier models it tested attempted to cheat on its cyber evaluations
DefenseEvaluation reportUK AI Security Institute, OpenAI, Anthropic

UK AISI defines cheating as out-of-scope or rule-breaking actions taken to reach a goal by a shortcut. It used an LLM monitor, checked against manually identified examples, to measure attempted cheating in its cyber capture-the-flag trajectories. All five models tested (GPT-5.4, GPT-5.5, GPT-5.6 Sol, Claude Opus 4.7 and Claude Mythos Preview) attempted to cheat in roughly 8-14% of runs. Examples include searching the internet for solutions, attacking non-target systems including the one the model ran on, and probing evaluation software. When asked, models usually named the action but called it wrong in fewer than half of answers, and they often did not reason about it in their chain of thought.

Jul 16, 2026
Cost-aware evaluation finds defensive SOC agents do not scale with compute like offensive CTF agents
DefensePaper

Researchers evaluate security agents at fixed cost levels on offensive Cybench challenges and defensive Splunk BOTS v1 investigations, splitting spend into inference and tool use. They find offensive success rises with test-time compute, while defensive investigation depends more on disciplined tool use and telemetry navigation, and argue benchmarks should report cost and operational fit alongside success.

Jul 8, 2026
FakeLab study finds distributed attacks harder for per-agent monitors to catch
DefensePaperMATS Research, UK AI Security Institute

MATS Research authors, one also affiliated with the UK AI Security Institute, evaluate distributed attacks in FakeLab, a synthetic lab codebase where agents work sequentially and monitors inspect individual commits. Splitting a shared attack reduces detection in tested settings, and an unmonitored explicit planner increases end-to-end attack completion as an upper bound on coordination. Results concern deliberately adversarial agents and selected monitor configurations, not observed collusion or measured deployment risk.

Jul 2, 2026
UK AISI finds agent evaluations understate cyber capability without accounting for test-time compute
DefenseEvaluation reportUK AI Security Institute

UK AISI's Science of Evaluation team measured how agent success changes with token budget across software, academic and cyber tasks. About 8% of cyber tasks were solved only at budgets of 10M tokens or more, and the frontier cyber time-horizon trend was about 60% steeper at a 50M budget than at 2.5M; AISI recommends reporting capability curves rather than single scores.

May 2026
May 29, 2026
OpenAI publishes a playbook on harness choice and validity checks for third-party evaluations
DefenseGuidanceOpenAI, UK AI Security Institute, METR

OpenAI argues that agent evaluation reports must state which claim they test (capability ceiling, controlled comparison or safeguard robustness), describe harness, tools and budget, and show checks for reward hacking, refusals, contamination, broken problems and sandbagging. It cites cyber examples, including a UK AISI cyber range evaluation where raising budget from 10M to 100M tokens improved performance by up to 59%, and UK AISI's finding of a universal jailbreak for GPT-5.5 cyber safeguards using a custom harness.

May 13, 2026
ExploitBench grades AI exploit development as a 16-step capability ladder on V8 bugs
CapabilityBenchmarkCarnegie Mellon University, Bugcrowd

Carnegie Mellon researchers released ExploitBench, which scores exploitation progress on 41 V8 JavaScript-engine vulnerabilities across 16 flags from reaching the bug through arbitrary read/write, control-flow hijack and code execution. The paper reports that public models routinely reach and crash vulnerable code but rarely achieve arbitrary code execution, while one private frontier model succeeded on roughly half of cases.

Findings

Research openings

Benchmarks and tools