Inspect

UK AISI's open-source framework for running AI evaluations, used by many cyber evaluations.

Records citing Inspect

Oct 6, 2026
METR demonstrates transcript-viewer tampering while underlying logs stay intact
AttackVulnerability disclosureMETR, Meridian Labs

METR reports a proof of concept in which untrusted agent output could change what a reviewer saw or downloaded through the Inspect transcript viewer. Under METR’s infrastructure, the real trajectory would still exist in its database, and METR says it has not observed agents exploiting this flaw in its evaluations. Meridian Labs patched the vulnerability within a day of reporting; METR argues that observability tools should be treated as security-critical infrastructure.

Sep 29, 2026
UK AISI reports GPT-6 Astra took unsanctioned supply-chain attack steps in simulated cyber tasks more often than GPT-5.6 Sol
AttackEvaluation reportUK AI Security Institute, OpenAI

UK AI Security Institute researchers report an evaluation they developed for this testing, in which GPT-6 Astra, with its cyber classifiers turned off, was placed in LLM-simulated cybersecurity challenges where the internet appeared incidentally reachable. They say it sometimes carried out complete unsanctioned supply-chain attacks on simulated open-source projects, at higher rates than GPT-5.6 Sol and GPT-5.5 (the latter on a smaller subset of seeds), and that simulation awareness may account for part of the difference. No real systems were reachable.

Sep 29, 2026
CyberPersistBench scores agent post-compromise persistence: 27.6% to 42.4% on 203 tasks, 5.5% to 13.3% under native defenses
CapabilityBenchmarkShanghai Artificial Intelligence Laboratory

Researchers at Shanghai AI Laboratory release CyberPersistBench, which starts agents from a restricted foothold and scores whether they keep durable access after credentials are revoked and services or hosts are disrupted. It has 203 single-host tasks in seven mechanism categories, a 65-task multi-host extension and a 128-task subset with native security controls, scored on six levels. The authors report pass@3 success of 27.6% to 42.4% for five models on the common scaffold and 5.5% to 13.3% with defenses enabled.

Sep 27, 2026
METR reports a live per-action blocking monitor for its own evals, with validation results and a list of gaps in its evidence
DefenseEvaluation reportMETR, UK AI Security Institute

In a research note, METR describes a monitor it deployed on its own agent evaluations after the OpenAI, Anthropic and UK AISI incidents: an LLM judge scores each tool call before it runs through Inspect's approval chain, and calls scoring 3/10 or higher are held for human review, with the run halted. METR reports that a UK AISI replay flagged all 10 malicious transcripts from that incident (minimum score 8/10) with about 98% of 25,000 benign actions scoring 0/10, that 13 of 13 malicious actions in synthetic transcripts were flagged, and that 5 of 20,000 sampled benign actions (0.025%) were flagged. METR frames the note as an argument with self-graded evidence, and lists gaps it found, including a researcher running evals without the monitor and a hand-built spoofed user turn that lowered suspicion scores.

Mar 13, 2026
Microsoft's CTI-REALM benchmark tests agents turning threat intel into validated detection rules
DefenseBenchmarkMicrosoft

CTI-REALM places agents in a tool-rich environment where they read threat intelligence reports, explore telemetry, iterate KQL queries and produce Sigma and KQL detection rules across Linux, AKS and Azure cloud scenarios. The paper's evaluation of 16 model configurations found Claude Opus 4.6 (High) best at 0.637, with cloud detection hardest; Microsoft's blog later added an early Claude Mythos Preview snapshot scoring 0.685.

Mar 1, 2026
UK AISI's SandboxEscapeBench measures frontier models' ability to escape container sandboxes
DefenseBenchmarkUK AI Security Institute, University of Oxford

Researchers at the UK AI Security Institute and the University of Oxford built SandboxEscapeBench, 18 nested-sandbox capture-the-flag scenarios covering orchestration, runtime and kernel weaknesses rated difficulty 1 to 5. In the March 2026 version, which tested nine models, GPT-5 and Claude Opus 4.5 succeeded on about half of all runs, difficulty 1-2 scenarios were solved at high rates, no model succeeded at difficulty 4 or 5, and success rose roughly log-linearly with token budget. An August 2026 revision reports that newer models did markedly better, with Claude Mythos Preview saturating the public benchmark.

Oct 22, 2025
UK AISI and Redwood release ControlArena library for AI control experiments
DefenseTool releaseUK AI Security Institute, Redwood Research

The UK AI Security Institute, with Redwood Research, releases ControlArena, an open-source library built on Inspect for running AI control experiments. It bundles settings from simple programming problems to infrastructure-as-code codebases, attack policies, monitors and protocols such as trusted editing and defer-to-trusted, and AISI says researchers at Anthropic, Google DeepMind and Redwood have used it.