Adversarial tests, not deployment incident rates. METR’s evaluation-monitor work includes a hand-built spoofed-user-turn evasion. FakeLab adds distributed coding attacks under per-commit diff monitoring; its overt unmonitored planner is an upper bound on coordination and its strongest low-success operating points depend on costly monitoring and unusually detailed task specifications.
Corroborated: Supported by at least two independent sources.
Evidence
Model-elicited attacks are mode-collapsed and overstate catch rates.
Suspicion scores for covert API-key exposure driven down to 3 out of 100.
METR tests a per-action monitor for evaluations and reports evasion by spoofed user turns in a hand-built attempt.
In synthetic FakeLab, sequential agents split deliberately malicious work across commits reviewed independently. An unmonitored planner is an upper bound on coordination; stronger monitors lower undetected success at compute and false-positive costs.
Edited real agent transcripts: 17 attacks that evaded a proxy monitor also evaded a held-out action-only monitor from a frontier developer, and one annotator confirmed 6 as valid. The edited actions were never executed.
METR reports that a three-week review with Redwood Research found coverage gaps and possible ways to avoid or subvert OpenAI’s internal monitoring as it stood on 2026-07-21; METR has not evaluated OpenAI’s remediation claims.
How it relates to other findings
Select a finding to see how it relates to others. Arrows point from the newer finding to the one it supports, qualifies, contests, or supersedes.
- This finding qualifies OpenAI reports its internal coding-agent monitor matched every staff escalation, and OpenAI and Google DeepMind report that most flags reflect overeagerness or mistakes rather than adversarial intent.Catch rates against ordinary mistakes say little about adversarial evasion.
Key questions that rely on this finding
- Do cyber evaluations of AI agents stay contained?Not reliably. Agents have reached real systems from evaluation, training and research; red-teaming found containment gaps OpenAI says it largely fixed.
- Can AI agents defend and oversee systems on their own?Not reliably. Some designs improve benchmark triage, but autonomous defense lacks operational validation and production oversight of agents has documented gaps.
Status history
- 2025-06-17ReportedSHADE-Arena: some hidden sabotage goes undetected. · record
- 2026-05-10CorroboratedMonitoringBench shows weak attacks overstate monitor performance. · record
- 2026-09-25CorroboratedcorrectionSHADE-Arena and MonitoringBench share an author and count once. Corroboration rests on UK AISI's Control Red Team, which drove suspicion scores for covert API-key exposure to 3/100 on Google DeepMind and Anthropic monitors. · record