Adaptive red-teaming

NIST announced a paper by Apostol Vassilev in IEEE Security & Privacy arguing, by extension of Gödel's incompleteness results, that no finite set of guardrails can be universally robust against adversarial prompts. NIST recommends a continuous monitor-and-update model: ongoing red teaming, continuous guardrail updates, and operational resilience to limit impact and recover.

The Frontier Model Forum described security practices for AI agents: limiting agent actions and resource access to what is strictly necessary, sandboxing with filesystem scope and egress policies, deterministic controls outside the model's reasoning loop, confirmation before high-stakes actions, and audit logs. It also covers layered prompt injection defenses, and names adaptive least privilege and extending identity standards such as OAuth 2.0 to agents as promising or developing areas.

OpenAI argues that agent evaluation reports must state which claim they test (capability ceiling, controlled comparison or safeguard robustness), describe harness, tools and budget, and show checks for reward hacking, refusals, contamination, broken problems and sandbagging. It cites cyber examples, including a UK AISI cyber range evaluation where raising budget from 10M to 100M tokens improved performance by up to 59%, and UK AISI's finding of a universal jailbreak for GPT-5.5 cyber safeguards using a custom harness.

Lee, Bae and Yun show that existing CTF benchmarks can be solved by retrieving published writeups when agents have web search, and propose CTFusion, which evaluates agents on live CTF competitions through an MCP server on the CTFd platform. They test 3 LLMs and 2 agent designs across 5 live CTF events.

Jotautaitė, Martinez, Matthews and Tracy (two listed as independent, Tracy at Redwood Research) build a semi-automated red-teaming pipeline in the BashArena control setting, where agents covertly pursue side tasks such as firewall backdoors, privilege escalation, credential exfiltration and malware download. MonitoringBench's 2,644 attack trajectories lowered a Claude Opus 4.5 monitor's catch rate at 1% false positives from 94.9% on directly elicited attacks to 60.3% on refined ones.

Microsoft researchers red-teamed an internal platform of over 100 always-on LLM agents that represent different people and interact through forums, messages and a marketplace. They describe four network-level failure modes: self-propagating messages, amplification of false claims, capture of reputation and verification systems, and hard-to-trace flows through unwitting intermediaries. A small share of agents spontaneously adopted protective behaviors that spread through the network.

A competition run by Gray Swan with NIST's CAISI, the UK AI Security Institute and frontier labs asked 464 participants to craft indirect prompt injections that make tool-use, coding and computer-use agents take harmful actions while hiding any sign of compromise from the user. Participants made 272,000 attempts against 13 frontier models, yielding 8,648 successes; per-model success ranged from 0.5% (Claude Opus 4.5) to 8.5% (Gemini 2.5 Pro), and at least one attack succeeded against every model.

A systematization-of-knowledge paper by organizers and competitors analyzes AIxCC's design, the seven finalist architectures and results beyond the scoreboard. It reports that system stability and accuracy penalties decided rankings, that LLM-based systems found vulnerabilities a fuzzing baseline missed, and that among patches passing all automatic validation, manual review found semantic errors in 38-46% from baseline agents; the top two systems had 83.8% and 79.2% competition-scored patch accuracy.

Anthropic's Claude Opus 4.6 system card reports prompt injection attack success separately for tool use (Gray Swan's ART benchmark), coding and computer use (Gray Swan's Shade adaptive attacker), and browser use (an internal Best-of-N attacker), with and without extra safeguards and across different attempt budgets. For Opus 4.6, results range from 0% in coding to 85.7% in computer use with 200 attempts and no safeguards (78.6% with extended thinking). Anthropic notes that, unlike earlier Claude models, extended thinking increased ART attack success for this model.

OpenAI describes an LLM-based attacker trained end-to-end with reinforcement learning that searches for prompt injections able to steer the Atlas browser agent through long, multi-step harmful workflows, and a rapid response loop that adversarially trains new agent checkpoints against discovered attacks. OpenAI says the attacker found strategies absent from human red-teaming and external reports, and states that prompt injection is unlikely ever to be fully solved.

Anthropic describes three defenses for browser use: reinforcement learning on injected web content, classifiers that scan untrusted content, and human red-teaming including external arena-style challenges. Against an internal adaptive Best-of-N attacker with 100 attempts per environment, its chart shows Claude Opus 4.5 with the improved Claude for Chrome safeguards at 1.4% attack success (8.0% for the model alone), which the post rounds to 1%. Anthropic says this still represents meaningful risk.

OpenAI describes prompt injection as social engineering aimed at AI agents and lists its layered defenses: instruction-hierarchy safety training, automated red-teaming, AI-based monitors that can be updated quickly, sandboxing of code-running tools, link approval, confirmation before sensitive steps, logged-out mode in Atlas, and a watch mode on sensitive sites that pauses the agent if the user leaves the tab. It cites thousands of hours of prompt-injection-focused red teaming and a bug bounty, and says it has not yet seen significant attacker adoption of the technique.

Nasr, Carlini, Tramèr and 11 co-authors apply gradient, reinforcement learning, search and human red-teaming attacks to 12 published defenses. Most defenses originally reported near-zero attack success, but the adaptive attacks exceed 90% success against most, and human red-teamers succeeded on every challenge in the subset of defenses they were given.

A systematic review in ACM Computing Surveys covers autonomous blue- and red-team agents and cyber operations environments, and proposes criteria for judging whether autonomous network defence is ready for real deployment. It identifies gaps in explainability, continual learning under evolving threats, and realistic training environments.

Announcing a limited pilot of Claude in Chrome, Anthropic reports red-teaming with 123 test cases across 29 attack scenarios. Attack success in autonomous mode was 23.6% without new mitigations and 11.2% with them; on a separate set of browser-specific attacks, mitigations reduced success from 35.7% to 0%.

Zou and colleagues (Gray Swan and collaborators; Anthropic describes the resulting benchmark as developed with the UK AI Security Institute) report a public red-teaming competition with 1.8 million prompt-injection attacks against 22 frontier agents in 44 deployment scenarios, producing over 60,000 successful policy violations. From these they build the Agent Red Teaming (ART) benchmark and find nearly all agents break within 10 to 100 queries for most behaviors, with high transfer and little correlation between robustness and model size or capability.

Microsoft researchers and collaborators report on LLMail-Inject, a public challenge in which participants tried to inject instructions into emails to trigger unauthorized tool calls by an LLM email assistant protected by various defenses. The released dataset contains 208,095 unique attack submissions from 839 participants across multiple defenses, models and retrieval configurations.

Shi and colleagues describe Google DeepMind's continuous adaptive-attack evaluation of Gemini against indirect prompt injection in tool-use settings. On Gemini 2.0, adaptive attacks generally matched or beat non-adaptive ones against eight baseline defenses, reaching 98.4% against in-context learning and 82.4% against spotlighting, while a warning defense and a user-instruction classifier held (at most 10.8% and 3.0%). Adversarial fine-tuning for Gemini 2.5 lowered but did not eliminate attack success.

NIST's AI safety institute technical staff (renamed the Center for AI Standards and Innovation in June 2025) extended AgentDojo and red-teamed agents built on the upgraded Claude 3.5 Sonnet. On held-out Workspace tasks, attack success rose from 11% for the strongest baseline attack to 81% for the strongest newly developed attack, and across five injection tasks from 57% to 80% when each attack was tried 25 times. The team released an Inspect-based AgentDojo port and ran the red teaming with the UK AI Security Institute.