Methods/Evaluation method

Compute-scaled evaluation

Measuring capability across token and compute budgets instead of at one fixed budget.

9 records4 capability5 defense3 findings (3 measured)First recorded 2026-02assistant-drafted

How it works

The same tasks are run at several budgets; capability that appears only at higher budgets would be missed by a single low cap.

What we know

2 reported, 1 qualified

Records over time

RangeLanes
9 of 9 records in view

Use the arrow keys to move between records, Home and End to jump to the first and last, and Enter to select one.

Agents find real bugsAgents in real operationsGated capability, incidents in the labAttackCapabilityDefensePolicyJan 25Jul 25Jan 26Jul 26
Full record · drag to choose a range
2026

Select a mark to read the record. Mark size shows editorial significance. Hollow marks are dated to the month. Era bands are editorial labels.

Records in view

9 records · newest first
Nov 2026
Nov 9, 2026
DeepMind tests adaptive injection swarms against synthetic tool-call defenses
CapabilityPaperGoogle DeepMind

A Google DeepMind preprint tests collaborative attacker agents on a synthetic instruction-hierarchy benchmark with randomly selected, non-malicious tool-call goals. Stronger attackers and larger query budgets expose failures that static tests miss. The experiments isolate model behavior and do not measure successful real-world data theft or a complete deployed defense stack.

Nov 7, 2026
CyberBinGym separates binary vulnerability recovery from working exploitation
CapabilityBenchmark

The CyberBinGym preprint evaluates agents given stripped vulnerable and patched binaries for 100 tasks from 53 projects. On the non-hardened baseline, Codex with GPT-5.6 recovers vulnerabilities on 88% of tasks and completes exploitation on 20%. The known-vulnerability patch-gap tests use trusted-access models with safeguards disabled; hardened configurations are evaluated separately.

Sep 2026
Sep 29, 2026
Anthropic assesses open-weight GLM-5.3: exploit results near Mythos Preview, safeguards bypassed in 64% to 100% of simulated trials
CapabilityEvaluation reportAnthropic, US Center for AI Standards and Innovation, Moonshot AI

Anthropic's Frontier Red Team reports that Zhipu AI's open-weight GLM-5.3 developed end-to-end exploits in 50 of 410 ExploitBench attempts, against 56 of 410 for Claude Mythos Preview, and full control-flow hijacks in 4% of trials on a 100-task subset of Anthropic's internal binary-exploitation benchmark, against 6% for Mythos Preview. In a simulated harmful-request test, Anthropic reports the model's refusals were bypassed in 64% to 100% of trials using a cover story, prefilled reasoning, or an abliterated copy of the weights, techniques it says did not work on safeguarded Claude models. All figures are Anthropic's own and were produced on setups the post describes only in part.

Sep 22, 2026
Claude Opus 5.5 system card reports cyber results at or above Mythos 5.1 and a three-stage cyber safeguard
CapabilitySystem cardAnthropic, Irregular, US Center for AI Standards and Innovation

Anthropic's Claude Opus 5.5 system card reports that, evaluated through the API with its cyber safeguards off, the model meets or exceeds Claude Mythos 5.1 and Claude Opus 5 on every cyber evaluation it includes (ExploitBench, CyScenarioBench, a rewritten Binary Exploitation Benchmark and ExploitGym). Anthropic places the model in the lower category of its Frontier Compliance Framework and says it sees no indication of novel offensive capability. It deployed the model with a three-stage cyber safeguard (an activation probe and two classifiers) that blocks vulnerability discovery in compiled binaries, and reports no evidence of a critical-severity jailbreak from internal and contracted external red teaming (the card gives no CAISI results).

Jul 2026
Jul 16, 2026
Cost-aware evaluation finds defensive SOC agents do not scale with compute like offensive CTF agents
DefensePaper

Researchers evaluate security agents at fixed cost levels on offensive Cybench challenges and defensive Splunk BOTS v1 investigations, splitting spend into inference and tool use. They find offensive success rises with test-time compute, while defensive investigation depends more on disciplined tool use and telemetry navigation, and argue benchmarks should report cost and operational fit alongside success.

Jul 2, 2026
UK AISI finds agent evaluations understate cyber capability without accounting for test-time compute
DefenseEvaluation reportUK AI Security Institute

UK AISI's Science of Evaluation team measured how agent success changes with token budget across software, academic and cyber tasks. About 8% of cyber tasks were solved only at budgets of 10M tokens or more, and the frontier cyber time-horizon trend was about 60% steeper at a 50M budget than at 2.5M; AISI recommends reporting capability curves rather than single scores.

May 2026
May 29, 2026
OpenAI publishes a playbook on harness choice and validity checks for third-party evaluations
DefenseGuidanceOpenAI, UK AI Security Institute, METR

OpenAI argues that agent evaluation reports must state which claim they test (capability ceiling, controlled comparison or safeguard robustness), describe harness, tools and budget, and show checks for reward hacking, refusals, contamination, broken problems and sandbagging. It cites cyber examples, including a UK AISI cyber range evaluation where raising budget from 10M to 100M tokens improved performance by up to 59%, and UK AISI's finding of a universal jailbreak for GPT-5.5 cyber safeguards using a custom harness.

Mar 2026
Mar 1, 2026
UK AISI's SandboxEscapeBench measures frontier models' ability to escape container sandboxes
DefenseBenchmarkUK AI Security Institute, University of Oxford

Researchers at the UK AI Security Institute and the University of Oxford built SandboxEscapeBench, 18 nested-sandbox capture-the-flag scenarios covering orchestration, runtime and kernel weaknesses rated difficulty 1 to 5. In the March 2026 version, which tested nine models, GPT-5 and Claude Opus 4.5 succeeded on about half of all runs, difficulty 1-2 scenarios were solved at high rates, no model succeeded at difficulty 4 or 5, and success rose roughly log-linearly with token budget. An August 2026 revision reports that newer models did markedly better, with Claude Mythos Preview saturating the public benchmark.

Feb 2026
Feb 7, 2026
AIxCC SoK finds stability decided results and many validated AI patches were still semantically wrong
DefensePaperGeorgia Institute of Technology, Texas A&M University, DARPA

A systematization-of-knowledge paper by organizers and competitors analyzes AIxCC's design, the seven finalist architectures and results beyond the scoreboard. It reports that system stability and accuracy penalties decided rankings, that LLM-based systems found vulnerabilities a fuzzing baseline missed, and that among patches passing all automatic validation, manual review found semantic errors in 38-46% from baseline agents; the top two systems had 83.8% and 79.2% competition-scored patch accuracy.

All records