Atlas/planner · policy

Agent model

The model making decisions: its safeguards, refusals, and susceptibility to jailbreaks.

2 attacks and incidents23 defense records5 findings2 measured defense findings

Attacks and incidents

May 11, 2026

Google Threat Intelligence reports the first criminal zero-day exploit it believes was AI-developed, disrupted before planned mass use

AttackMisuse reportGoogle Threat Intelligence Group, UNC6780 (TeamPCP)

Feb 23, 2023

Researchers define indirect prompt injection against LLM-integrated apps including Bing Chat

AttackPaperCISPA Helmholtz Center for Information Security, sequire technology

Defense research

Sep 2, 2026

Google releases Gemini 3.8 Flash Cyber for trusted defenders, emphasizing automated patching

DefenseTool releaseGoogle, Google DeepMind, Collinear

Jun 18, 2026

Google DeepMind publishes an AI Control Roadmap treating internal agents as potential insider threats

DefenseFrameworkGoogle DeepMind

May 22, 2026

Glasswing update: over 10,000 high-severity bugs found, but only 75 of 530 disclosed OSS bugs patched

DefenseProgramAnthropic, Cloudflare, Mozilla

Mar 16, 2026

CAISI, UK AISI and Gray Swan competition finds concealed indirect injections succeed on all 13 frontier models

DefensePaperGray Swan AI, US Center for AI Standards and Innovation, UK AI Security Institute

Mar 10, 2026

OpenAI releases IH-Challenge RL dataset and reports instruction hierarchy gains on injection benchmarks

DefenseDatasetOpenAI

Feb 5, 2026

Claude Opus 4.6 system card reports prompt injection rates by surface, attempts and safeguards

DefenseSystem cardAnthropic, Gray Swan AI

Jan 9, 2026

Anthropic's next-generation Constitutional Classifiers cut overhead to about 1% using probe cascades

DefensePaperAnthropic

Dec 22, 2025

OpenAI hardens ChatGPT Atlas with an RL-trained automated prompt injection attacker

DefenseFrameworkOpenAI

Nov 24, 2025

Anthropic reports 1.4% prompt injection success for Claude Opus 4.5 with improved Chrome extension safeguards

DefenseEvaluation reportAnthropic, Gray Swan AI

Oct 10, 2025

'The Attacker Moves Second': adaptive attacks bypass 12 published jailbreak and injection defenses

DefensePaperarXiv

Oct 3, 2025

Anthropic says it trained Claude Sonnet 4.5 for defensive vulnerability finding and patching

DefensePaperAnthropic, HackerOne, CrowdStrike

Jul 28, 2025

Large public competition finds all 22 tested frontier agents vulnerable to prompt injection

DefenseBenchmarkGray Swan AI, UK AI Security Institute

Jun 17, 2025

SHADE-Arena measures whether agents can pursue hidden harmful side tasks without being caught by monitors

DefenseBenchmarkAnthropic, Redwood Research

Jun 10, 2025

Researchers from labs and industry publish design patterns for provably injection-resistant agents

DefenseFrameworkInvariant Labs, IBM, ETH Zurich

May 20, 2025

Google DeepMind reports lessons from continuously attacking Gemini with adaptive prompt injections

DefensePaperGoogle DeepMind

May 6, 2025

Meta releases LlamaFirewall guardrails with PromptGuard 2 and AlignmentCheck for agents

DefenseTool releaseMeta

Mar 24, 2025

Google DeepMind's CaMeL defeats prompt injections by design with capability-based control and data flow

DefensePaperGoogle DeepMind, Google, ETH Zurich

Feb 3, 2025

Anthropic introduces Constitutional Classifiers against universal jailbreaks

DefensePaperAnthropic

Oct 7, 2024

SecAlign uses preference optimization to train LLMs against prompt injection

DefensePaperMeta, UC Berkeley

Apr 19, 2024

OpenAI trains models to prioritize privileged instructions via an instruction hierarchy

DefensePaperOpenAI

Mar 20, 2024

Microsoft researchers propose spotlighting to mark untrusted input against indirect prompt injection

DefensePaperMicrosoft

Feb 9, 2024

StruQ proposes separating prompts and data channels to defend against prompt injection

DefensePaperUC Berkeley

Dec 12, 2023

Redwood Research introduces AI control protocols for safety despite intentional subversion

DefensePaperRedwood Research

Findings

Jan 17, 2025

Attackers who adapt to a defense defeat most published prompt-injection defenses that reported near-zero success against static attacks.

Reportedmeasured4 evidence records2 linked findings

Jul 28, 2025

More capable models are not reliably more robust to prompt injection, and some are more susceptible.

Corroboratedmeasured3 evidence records

Feb 3, 2025

Anthropic reports that classifier guards cut automated jailbreak success from 86% to 4.4%; a 2025 public demo still yielded one universal jailbreak, though none has been reported against its 2026 successor.

Reportedreported2 evidence records

May 20, 2025

Frontier labs' layered defenses reduce prompt injection in tool-use, browser and computer-use agents but do not eliminate it.

Qualifiedreported5 evidence records2 linked findings

Jul 22, 2025

Limits that agents are only told about, or that block one tool while others remain available, do not reliably stop them: a declared code freeze and an ignore-file setting were both bypassed.

Corroboratedobserved2 evidence records1 linked finding

Capability, policy, and guidance

Jul 23, 2026

UK AISI and US CAISI jointly assess Kimi K3 cyber capability as trailing US frontier models

CapabilityEvaluation reportUK AI Security Institute, US Center for AI Standards and Innovation, Moonshot AI

Jul 2, 2026

Anthropic proposes Cyber Jailbreak Severity scale with Glasswing partners

PolicyFrameworkAnthropic

Jun 30, 2026

US lifts export controls on Fable 5 and Mythos 5; Anthropic redeploys with new cyber classifier

PolicyRegulationAnthropic, US Department of Commerce, US Center for AI Standards and Innovation

Jun 12, 2026

US export-control directive forces Anthropic to suspend Fable 5 and Mythos 5 over safeguard bypass

PolicyRegulationUS Department of Commerce, Anthropic

Jun 9, 2026

NIST scientist argues no finite guardrail set is robust to adversarial prompts, urges continuous updates

PolicyGuidanceNIST

Jun 2, 2026

Executive Order 14409 creates classified cyber benchmarking for covered frontier models and a clearinghouse

PolicyRegulationThe White House, US Department of the Treasury, NSA

May 1, 2026

CAISI evaluation finds DeepSeek V4 Pro trails US frontier models by about eight months

CapabilityEvaluation reportUS Center for AI Standards and Innovation, DeepSeek, NIST

Apr 17, 2026

DeepMind Frontier Safety Framework v3.1 raises security for its cyber critical capability level

PolicyFrameworkGoogle DeepMind

Feb 24, 2026

Anthropic RSP v3.0 rewrite adds risk reports and roadmaps; policy text does not name cyber

PolicyFrameworkAnthropic

Feb 13, 2026

Frontier Model Forum report sets out shared cyber thresholds for frontier AI safety frameworks

PolicyFrameworkFrontier Model Forum

Sep 30, 2025

CAISI evaluation finds DeepSeek models lag US models on cyber tasks and are far easier to hijack

CapabilityEvaluation reportUS Center for AI Standards and Innovation, NIST, DeepSeek

Sep 29, 2025

California SB 53 requires frontier AI frameworks covering autonomous cyberattack risk and incident reporting

PolicyRegulationState of California

Sep 22, 2025

Google DeepMind Frontier Safety Framework v3 retains Cyber Uplift Level 1 critical capability level

PolicyFrameworkGoogle DeepMind

Aug 2, 2025

EU AI Act obligations for general-purpose AI model providers enter into application

PolicyRegulationEuropean Commission

Jul 10, 2025

EU GPAI Code of Practice Safety and Security chapter lists cyber offence as a specified systemic risk

PolicyFrameworkEuropean Commission

May 22, 2025

Anthropic activates ASL-3 deployment and security protections for Claude Opus 4

CapabilityThresholdAnthropic

Apr 15, 2025

OpenAI Preparedness Framework v2 sets High and Critical cybersecurity capability thresholds

PolicyFrameworkOpenAI

Feb 6, 2025

Cloud Security Alliance publishes MAESTRO seven-layer threat modeling framework for agentic AI

PolicyFrameworkCloud Security Alliance

Jan 15, 2025

NIST second draft of AI 800-1 on dual-use foundation model misuse adds cybersecurity appendix

PolicyGuidanceNIST, US Center for AI Standards and Innovation

Oct 15, 2024

Anthropic RSP v2 lists cyber operations as a capability under ongoing assessment, not a threshold

PolicyFrameworkAnthropic

Apr 15, 2024

NSA-led Five Eyes guidance on deploying externally developed AI systems securely

PolicyGuidanceNSA Artificial Intelligence Security Center, CISA, FBI

Jan 4, 2024

NIST publishes adversarial machine learning taxonomy covering direct and indirect prompt injection

PolicyStandardNIST

Nov 27, 2023

UK NCSC and US CISA publish multinational Guidelines for Secure AI System Development

PolicyGuidanceUK National Cyber Security Centre, CISA

Oct 30, 2023

US Executive Order 14110 names offensive cyber capability as a dual-use foundation model risk

PolicyRegulationThe White House

Related research openings

Do production prompt-injection defenses survive independent adaptive attack?