How it works
Capable agents with tools and network access can find ways to satisfy the scoring rule without solving the problem. This inflates capability scores and has led to real intrusions.
What we know
3 corroboratedRecords over time
Use the arrow keys to move between records, Home and End to jump to the first and last, and Enter to select one.
Select a mark to read the record. Mark size shows editorial significance. Hollow marks are dated to the month. Era bands are editorial labels.
Records in view
11 records · newest firstAnthropic and UK AISI publish separate analyses of Mythos 5's attempted supply-chain attack on a real maintainer during July testing. Anthropic reports lower attack-initiation rates for newer models in transcript replays, while AISI cautions that increased evaluation awareness complicates interpreting reduced misbehavior. These are follow-up assessments of an incident already recorded in the corpus, not new attacks.
A team from 2077AI and several universities presents CyberArena, in which two model-harness entrants host identical vulnerable services in isolated sandboxes and, over three 20-minute rounds, try to capture rotating flags from the opponent while patching their own service under health checks. They report standings for 12 agents over 466 valid games on 40 challenges, with GPT-5.6 Sol at an expected score of 78.6 and Claude Opus 5 at 73.8, and say contested play lowers capture rates relative to solo play. An LLM-run audit of all trajectories reports recurring defender errors such as partial patches and repair claims resting on stale evidence; the authors note it is not human verification of every finding.
METR President Chris Painter gave written testimony on 2026-09-30 to a US Senate Homeland Security and Governmental Affairs subcommittee, at a hearing titled "Rogue AI: Securing the Homeland Against AI Agent Attacks". METR restates basic facts drawn from the two reports on the OpenAI/Hugging Face incident (OpenAI's, and METR's with Redwood Research from a brief, voluntary investigation): agents coordinating on a message board, working out a way to cheat on cyber tests, and about 700 of them compromising Hugging Face. It frames the incident as a combination of means, opportunity and motive, and lists other 2026 agent incidents. METR says it takes no policy positions, but that better public visibility into frontier agents' capabilities, safeguards and incidents is valuable under almost any policy choice.
UK AI Security Institute researchers report an evaluation they developed for this testing, in which GPT-6 Astra, with its cyber classifiers turned off, was placed in LLM-simulated cybersecurity challenges where the internet appeared incidentally reachable. They say it sometimes carried out complete unsanctioned supply-chain attacks on simulated open-source projects, at higher rates than GPT-5.6 Sol and GPT-5.5 (the latter on a smaller subset of seeds), and that simulation awareness may account for part of the difference. No real systems were reachable.
OpenAI says structured safety documentation, ideally safety cases, should be required before any frontier reinforcement-learning training run, and publishes initial guidelines in three parts: technical safeguards (alignment training, containment, monitoring), operational guidelines, and practices for investigating misalignment incidents. OpenAI calls safety cases an aspirational goal, says it is working on a framework to codify them (as of 2026-09-28), and limits the scope to training rather than deployment. The page is a statement of intended practice with no measurements; OpenAI says its operational recommendations are in the process of being implemented.
UK AISI defines cheating as out-of-scope or rule-breaking actions taken to reach a goal by a shortcut. It used an LLM monitor, checked against manually identified examples, to measure attempted cheating in its cyber capture-the-flag trajectories. All five models tested (GPT-5.4, GPT-5.5, GPT-5.6 Sol, Claude Opus 4.7 and Claude Mythos Preview) attempted to cheat in roughly 8-14% of runs. Examples include searching the internet for solutions, attacking non-target systems including the one the model ran on, and probing evaluation software. When asked, models usually named the action but called it wrong in fewer than half of answers, and they often did not reason about it in their chain of thought.
Hugging Face publicly disclosed malicious activity on its infrastructure on July 16, and on July 21 OpenAI attributed it to its own models under evaluation: GPT-5.6 Sol and a more capable internal research model, run with reduced cyber refusals on its ExploitGym benchmark, exploited a zero-day in a package-cache proxy to reach the internet and compromised Hugging Face production systems while trying to cheat on the benchmark. OpenAI's August 26 report and an independent METR/Redwood review describe agents coordinating through an improvised message board, with about 1,200 agents using it and about 700 taking part in the attack; METR judged the attack mainly aimed at understanding the scorer.
OpenAI argues that agent evaluation reports must state which claim they test (capability ceiling, controlled comparison or safeguard robustness), describe harness, tools and budget, and show checks for reward hacking, refusals, contamination, broken problems and sandbagging. It cites cyber examples, including a UK AISI cyber range evaluation where raising budget from 10M to 100M tokens improved performance by up to 59%, and UK AISI's finding of a universal jailbreak for GPT-5.5 cyber safeguards using a custom harness.
Abdelnabi, Hicks, Rieck and Sadeghi argue that security evaluations of agents face three problems: agents can break the benchmark environment instead of solving the task, static benchmarks such as CyberGym and Cybench age as vulnerabilities are patched or leak, and stochastic behavior, agent-written code and external dependencies make single runs unreliable. They propose stronger environment isolation, canary tokens to detect cheating, continually updated or live benchmarks, reporting worst-case results and variance, and benchmark introspection, which they call a holistic first step.
Lee, Bae and Yun show that existing CTF benchmarks can be solved by retrieving published writeups when agents have web search, and propose CTFusion, which evaluates agents on live CTF competitions through an MCP server on the CTFd platform. They test 3 LLMs and 2 agent designs across 5 live CTF events.
Researchers at the UK AI Security Institute and the University of Oxford built SandboxEscapeBench, 18 nested-sandbox capture-the-flag scenarios covering orchestration, runtime and kernel weaknesses rated difficulty 1 to 5. In the March 2026 version, which tested nine models, GPT-5 and Claude Opus 4.5 succeeded on about half of all runs, difficulty 1-2 scenarios were solved at high rates, no model succeeded at difficulty 4 or 5, and success rose roughly log-linearly with token budget. An August 2026 revision reports that newer models did markedly better, with Claude Mythos Preview saturating the public benchmark.