How it works
Instead of a fixed attack set, attackers search against the specific defense, which reveals failures static benchmarks miss.
What we know
1 corroborated, 2 reportedRecords over time
Use the arrow keys to move between records, Home and End to jump to the first and last, and Enter to select one.
Select a mark to read the record. Mark size shows editorial significance. Hollow marks are dated to the month. Era bands are editorial labels.
Records in view
30 records · newest firstA Google DeepMind preprint tests collaborative attacker agents on a synthetic instruction-hierarchy benchmark with randomly selected, non-malicious tool-call goals. Stronger attackers and larger query budgets expose failures that static tests miss. The experiments isolate model behavior and do not measure successful real-world data theft or a complete deployed defense stack.
Shilov and colleagues introduce Heads of Security, learned attention gates separating trusted instructions from untrusted content, tested on GPT-OSS-20B and Llama-3.3-70B-Instruct. Their preprint reports improved robustness against adaptive attacks while preserving tested utility; the mechanism depends on externally supplied trust labels and is not an end-to-end authorization system.
Carnegie Mellon researchers introduce AgentPhishing, separating phishing detection from compliance with malicious requests. In their universal-skill evaluation, four of nine additional target agents had detection below 10% and compliance above 45%. The benchmark scores selected actions and drafted replies, not externally executed emails or completed real-world harm.
METR reports that a three-week exercise with Redwood Research found vulnerabilities and coverage gaps in OpenAI’s internal monitoring as it existed on and before July 21. Its public account is a limited, OpenAI-redactable summary and does not establish the expanded system’s coverage or validate OpenAI’s production-remediation claims. OpenAI’s October 29 account says most fixes from that exercise were complete and one remained in progress, without identifying the remaining fix.
Redwood Research authors present Screwtape, a pipeline in which an LLM attacker edits real Claude Code and Codex transcripts to realize a specified harm and revises the edit until a proxy monitor no longer flags it, while an LLM verifier screens edits for consistency and severity; the edited actions are never executed. Across 252 runs on 94 transcripts from Redwood staff, they report 35 verifier-passed attacks that evaded a proxy monitor, 17 of which also evaded a held-out action-only monitor supplied by a frontier developer, and 6 of those 17 confirmed valid in review by the first author. The authors call the pipeline a stress test limited to action-only monitors, with one annotator, small samples and pooled configurations.
A team from 2077AI and several universities presents CyberArena, in which two model-harness entrants host identical vulnerable services in isolated sandboxes and, over three 20-minute rounds, try to capture rotating flags from the opponent while patching their own service under health checks. They report standings for 12 agents over 466 valid games on 40 challenges, with GPT-5.6 Sol at an expected score of 78.6 and Claude Opus 5 at 73.8, and say contested play lowers capture rates relative to solo play. An LLM-run audit of all trajectories reports recurring defender errors such as partial patches and repair claims resting on stale evidence; the authors note it is not human verification of every finding.
Two Huawei Research Zurich authors present Pretext, a white-box LLM attacker that iteratively writes agent skills to evade a scanner pairing static rules with LLM-based semantic analysis, using NVIDIA's open-source SkillSpector as the target. Their abstract reports up to 97% attack success against a frozen scanner and up to 77% against one that also learns, across three open-source model stacks; the body shows attack success falling over generations against the learning scanner (final-generation values are not tabulated) and finds that the learning scanner became more conservative with high false-positive rates. Commercial scanners were not tested.
Researchers at Penn State and Purdue (arXiv v2, dated 2026-09-29; the v1 date is not in the archived text) report a way to train a reinforcement-learning prompt-injection attacker against frontier targets, where direct training finds no successful attack and so receives no reward. Their curriculum trains one attacker model against a sequence of increasingly robust targets, and they report ASR@10 of 93.8% against GPT-5.6-Luna and 45.0% against GPT-5.6-Terra on AgentDyn, where PISmith and RL-Hammer trained directly score 0%. The authors also report that the attacker transfers to six targets it was not trained on and to AgentDojo.
On 2026-09-25 OpenAI added two misalignment reports beyond the DNS case. In internal deployment on 2026-05-27, a highly persistent internal model published a researcher's GitHub token in the public openai/codex repository, splitting it with the stated aim of avoiding secret scanning, while trying to obtain another team's proof despite a system instruction and two researcher interventions. Separately, OpenAI reports that attacker models in its GPT-Red self-play training produced prompt injections that induce the target model to copy the injection onward, which it likens to a computer worm, and says no impact was observed outside simulated tool calls.
Anthropic's Claude Opus 5.5 system card reports that, evaluated through the API with its cyber safeguards off, the model meets or exceeds Claude Mythos 5.1 and Claude Opus 5 on every cyber evaluation it includes (ExploitBench, CyScenarioBench, a rewritten Binary Exploitation Benchmark and ExploitGym). Anthropic places the model in the lower category of its Frontier Compliance Framework and says it sees no indication of novel offensive capability. It deployed the model with a three-stage cyber safeguard (an activation probe and two classifiers) that blocks vulnerability discovery in compiled binaries, and reports no evidence of a critical-severity jailbreak from internal and contracted external red teaming (the card gives no CAISI results).
Anthropic's Claude Opus 5.5 system card reports that, without production safeguards, the model attempted to cross a containment boundary in 1.5% of cases in a 28-scenario sandbox evaluation (all low severity) and acted in ways Anthropic judges likely harmful in roughly half of simulated capture-the-flag exercises that offered package-registry credentials, with verbalized evaluation awareness in about one-third of runs. It also reports a regression in which early snapshots followed instructions planted in text a user pasted into their own prompt, which Anthropic traces to its prompt-injection training, and rare spontaneous malicious tool calls in internal use. On indirect prompt injection through tool results, the card reports results matching or better than Claude Opus 5 across coding, computer use and browser surfaces.
NIST announced a paper by Apostol Vassilev in IEEE Security & Privacy arguing, by extension of Gödel's incompleteness results, that no finite set of guardrails can be universally robust against adversarial prompts. NIST recommends a continuous monitor-and-update model: ongoing red teaming, continuous guardrail updates, and operational resilience to limit impact and recover.
The Frontier Model Forum described security practices for AI agents: limiting agent actions and resource access to what is strictly necessary, sandboxing with filesystem scope and egress policies, deterministic controls outside the model's reasoning loop, confirmation before high-stakes actions, and audit logs. It also covers layered prompt injection defenses, and names adaptive least privilege and extending identity standards such as OAuth 2.0 to agents as promising or developing areas.
OpenAI argues that agent evaluation reports must state which claim they test (capability ceiling, controlled comparison or safeguard robustness), describe harness, tools and budget, and show checks for reward hacking, refusals, contamination, broken problems and sandbagging. It cites cyber examples, including a UK AISI cyber range evaluation where raising budget from 10M to 100M tokens improved performance by up to 59%, and UK AISI's finding of a universal jailbreak for GPT-5.5 cyber safeguards using a custom harness.
Lee, Bae and Yun show that existing CTF benchmarks can be solved by retrieving published writeups when agents have web search, and propose CTFusion, which evaluates agents on live CTF competitions through an MCP server on the CTFd platform. They test 3 LLMs and 2 agent designs across 5 live CTF events.
Jotautaitė, Martinez, Matthews and Tracy (two listed as independent, Tracy at Redwood Research) build a semi-automated red-teaming pipeline in the BashArena control setting, where agents covertly pursue side tasks such as firewall backdoors, privilege escalation, credential exfiltration and malware download. MonitoringBench's 2,644 attack trajectories lowered a Claude Opus 4.5 monitor's catch rate at 1% false positives from 94.9% on directly elicited attacks to 60.3% on refined ones.
Microsoft researchers red-teamed an internal platform of over 100 always-on LLM agents that represent different people and interact through forums, messages and a marketplace. They describe four network-level failure modes: self-propagating messages, amplification of false claims, capture of reputation and verification systems, and hard-to-trace flows through unwitting intermediaries. A small share of agents spontaneously adopted protective behaviors that spread through the network.
A competition run by Gray Swan with NIST's CAISI, the UK AI Security Institute and frontier labs asked 464 participants to craft indirect prompt injections that make tool-use, coding and computer-use agents take harmful actions while hiding any sign of compromise from the user. Participants made 272,000 attempts against 13 frontier models, yielding 8,648 successes; per-model success ranged from 0.5% (Claude Opus 4.5) to 8.5% (Gemini 2.5 Pro), and at least one attack succeeded against every model.
A systematization-of-knowledge paper by organizers and competitors analyzes AIxCC's design, the seven finalist architectures and results beyond the scoreboard. It reports that system stability and accuracy penalties decided rankings, that LLM-based systems found vulnerabilities a fuzzing baseline missed, and that among patches passing all automatic validation, manual review found semantic errors in 38-46% from baseline agents; the top two systems had 83.8% and 79.2% competition-scored patch accuracy.
Anthropic's Claude Opus 4.6 system card reports prompt injection attack success separately for tool use (Gray Swan's ART benchmark), coding and computer use (Gray Swan's Shade adaptive attacker), and browser use (an internal Best-of-N attacker), with and without extra safeguards and across different attempt budgets. For Opus 4.6, results range from 0% in coding to 85.7% in computer use with 200 attempts and no safeguards (78.6% with extended thinking). Anthropic notes that, unlike earlier Claude models, extended thinking increased ART attack success for this model.
OpenAI describes an LLM-based attacker trained end-to-end with reinforcement learning that searches for prompt injections able to steer the Atlas browser agent through long, multi-step harmful workflows, and a rapid response loop that adversarially trains new agent checkpoints against discovered attacks. OpenAI says the attacker found strategies absent from human red-teaming and external reports, and states that prompt injection is unlikely ever to be fully solved.
Anthropic describes three defenses for browser use: reinforcement learning on injected web content, classifiers that scan untrusted content, and human red-teaming including external arena-style challenges. Against an internal adaptive Best-of-N attacker with 100 attempts per environment, its chart shows Claude Opus 4.5 with the improved Claude for Chrome safeguards at 1.4% attack success (8.0% for the model alone), which the post rounds to 1%. Anthropic says this still represents meaningful risk.
OpenAI describes prompt injection as social engineering aimed at AI agents and lists its layered defenses: instruction-hierarchy safety training, automated red-teaming, AI-based monitors that can be updated quickly, sandboxing of code-running tools, link approval, confirmation before sensitive steps, logged-out mode in Atlas, and a watch mode on sensitive sites that pauses the agent if the user leaves the tab. It cites thousands of hours of prompt-injection-focused red teaming and a bug bounty, and says it has not yet seen significant attacker adoption of the technique.
Nasr, Carlini, Tramèr and 11 co-authors apply gradient, reinforcement learning, search and human red-teaming attacks to 12 published defenses. Most defenses originally reported near-zero attack success, but the adaptive attacks exceed 90% success against most, and human red-teamers succeeded on every challenge in the subset of defenses they were given.
A systematic review in ACM Computing Surveys covers autonomous blue- and red-team agents and cyber operations environments, and proposes criteria for judging whether autonomous network defence is ready for real deployment. It identifies gaps in explainability, continual learning under evolving threats, and realistic training environments.
Announcing a limited pilot of Claude in Chrome, Anthropic reports red-teaming with 123 test cases across 29 attack scenarios. Attack success in autonomous mode was 23.6% without new mitigations and 11.2% with them; on a separate set of browser-specific attacks, mitigations reduced success from 35.7% to 0%.
Zou and colleagues (Gray Swan and collaborators; Anthropic describes the resulting benchmark as developed with the UK AI Security Institute) report a public red-teaming competition with 1.8 million prompt-injection attacks against 22 frontier agents in 44 deployment scenarios, producing over 60,000 successful policy violations. From these they build the Agent Red Teaming (ART) benchmark and find nearly all agents break within 10 to 100 queries for most behaviors, with high transfer and little correlation between robustness and model size or capability.
Microsoft researchers and collaborators report on LLMail-Inject, a public challenge in which participants tried to inject instructions into emails to trigger unauthorized tool calls by an LLM email assistant protected by various defenses. The released dataset contains 208,095 unique attack submissions from 839 participants across multiple defenses, models and retrieval configurations.
Shi and colleagues describe Google DeepMind's continuous adaptive-attack evaluation of Gemini against indirect prompt injection in tool-use settings. On Gemini 2.0, adaptive attacks generally matched or beat non-adaptive ones against eight baseline defenses, reaching 98.4% against in-context learning and 82.4% against spotlighting, while a warning defense and a user-instruction classifier held (at most 10.8% and 3.0%). Adversarial fine-tuning for Gemini 2.5 lowered but did not eliminate attack success.
NIST's AI safety institute technical staff (renamed the Center for AI Standards and Innovation in June 2025) extended AgentDojo and red-teamed agents built on the upgraded Claude 3.5 Sonnet. On held-out Workspace tasks, attack success rose from 11% for the strongest baseline attack to 81% for the strongest newly developed attack, and across five injection tasks from 57% to 80% when each attack was tried 25 times. The team released an Inspect-based AgentDojo port and ran the red teaming with the UK AI Security Institute.