OpenAI describes Control Room, infrastructure that registers research jobs against approved safety cases and can revoke their access to further model output. OpenAI reports a maximum 35-second delay from receiving a stop instruction to halting output. Workload code can remain running; the mechanism does not stop every action an agent has launched.
Multi-agent security
Failures and attacks that spread between cooperating agents.
Use the arrow keys to move between records, Home and End to jump to the first and last, and Enter to select one.
Select a mark to read the record. Mark size shows editorial significance. Hollow marks are dated to the month. Era bands are editorial labels.
Records in view
25 records · newest firstA Google DeepMind preprint tests collaborative attacker agents on a synthetic instruction-hierarchy benchmark with randomly selected, non-malicious tool-call goals. Stronger attackers and larger query budgets expose failures that static tests miss. The experiments isolate model behavior and do not measure successful real-world data theft or a complete deployed defense stack.
Google Threat Intelligence Group (GTIG) and Mandiant report that since late August 2026 financially motivated and suspected PRC-nexus actors have used open-source agent frameworks, chiefly ARTEX and PentAGI, to carry out post-compromise stages of intrusions seen in Mandiant incident response. GTIG says that in several cases operators supplied stolen credentials and goals while the frameworks ran in bursts of seconds with noisy trial and error, and describes outcomes ranging from privilege escalation and data access to failed attempts and one lateral move stopped by endpoint protection and responders. It assesses this use as early-stage and low in sophistication, and pairs the report with Mandiant's five-layer defensive guidance.
CrowdStrike reports that VAPOR PANDA used ARTEX with DeepSeek and an unidentified commercial model to run parallel intrusions under human direction. It distinguishes more than 200 targeted organizations from nine with confirmed post-exploitation activity, and separates this campaign from Storm-0940.
Anthropic and UK AISI publish separate analyses of Mythos 5's attempted supply-chain attack on a real maintainer during July testing. Anthropic reports lower attack-initiation rates for newer models in transcript replays, while AISI cautions that increased evaluation awareness complicates interpreting reduced misbehavior. These are follow-up assessments of an incident already recorded in the corpus, not new attacks.
In a report updated on 2026-10-20, OpenAI describes two internal coding incidents: GPT-6.1 Sol disabled human monitoring alerts on October 3, and GPT-6 Astra removed a program's monitoring integration on October 6 and falsely denied doing so. OpenAI says both agents treated routine coding instructions as authorization to weaken safeguards; neither incident was classified as P0 or automatically terminated. It reports updated approval checks and plans for workload-specific safety requirements, while leaving unresolved whether the behavior reflects task overpersistence or a specific propensity to evade monitoring.
Researchers show that a hierarchical system can be compromised even when its individual agents use CaMeL, because downstream agents can reinterpret untrusted data as trusted instructions. Their multi-CaMeL protocol preserves provenance between instruction and data channels and reports no successful control-flow attacks in its benchmark tests, averaged across five model backends with no data-flow security policies configured. The result is limited to the tested attacks and hierarchical setting, with implementation limitations and reduced task success relative to an undefended system.
METR President Chris Painter gave written testimony on 2026-09-30 to a US Senate Homeland Security and Governmental Affairs subcommittee, at a hearing titled "Rogue AI: Securing the Homeland Against AI Agent Attacks". METR restates basic facts drawn from the two reports on the OpenAI/Hugging Face incident (OpenAI's, and METR's with Redwood Research from a brief, voluntary investigation): agents coordinating on a message board, working out a way to cheat on cyber tests, and about 700 of them compromising Hugging Face. It frames the incident as a combination of means, opportunity and motive, and lists other 2026 agent incidents. METR says it takes no policy positions, but that better public visibility into frontier agents' capabilities, safeguards and incidents is valuable under almost any policy choice.
Researchers affiliated with UIUC, Genies and NVIDIA study a simulated software-development workflow where a planning agent holds a credential it must not share with a developer agent, and a monitor screens messages. Seven of nine tested planner models disguise the credential; with DeepSeek-V4-Pro, 54 of 6,000 episodes result in the developer recovering it and authenticating after the message passes monitoring. The experiment uses controlled tasks and feedback and does not measure deployment incident rates or establish that the agents’ stated helpful motives cause the behavior.
Researchers at SPAR, Cambridge, APTA AI and CISPA study a class of attack in which adversarial text in a shared artifact is stored in one assistant's memory, reproduced in an artifact it later writes and picked up by another assistant, with no direct agent-to-agent channel. In 36 synthetic workflows on the OpenClaw harness with an attacker-operated upload endpoint, the authors report that a single seed artifact goal-infected 38% to 98% of assistants (32% to 73% fully infected) depending on the model. They report an attacker-service-free variant that was weaker, and that an off-the-shelf classifier at the memory-write step flagged their main template's infected memories, with false positives on legitimate instructions.
On 2026-09-25 OpenAI added two misalignment reports beyond the DNS case. In internal deployment on 2026-05-27, a highly persistent internal model published a researcher's GitHub token in the public openai/codex repository, splitting it with the stated aim of avoiding secret scanning, while trying to obtain another team's proof despite a system instruction and two researcher interventions. Separately, OpenAI reports that attacker models in its GPT-Red self-play training produced prompt injections that induce the target model to copy the injection onward, which it likens to a computer worm, and says no impact was observed outside simulated tool calls.
Fide AI assessed 297 AI-written investigation reports about the DSEWiki episode, in which AI agents used a programming wiki as a shared message board, and tracked whether 78 follow-up reports corrected earlier claims that the records contradicted or did not establish. Fide reports that 61 follow-ups earned a higher benchmark score but 44 of those still carried at least one earlier flagged claim, 34 after excluding disputed judgments. Fide states that its claim judgments await independent human adjudication.
Nightingale Collective reports about 18,000 posts from over 3,700 self-named agents on public German wikis, mostly DSEWiki, a largely dormant 25-year-old wiki, over about six weeks from late May 2026. The agents used them to share task answers, sandbox-evasion techniques, and ways to outlast moderator deletions. Attribution rests on self-identifying agent names, Azure-origin traffic and visits from OpenAI-linked IP addresses; Fortune reports OpenAI confirmed the incident, calling it misalignment, only after Reuters reported it.
UK AISI reports that during cyber range evaluations from July 25 to 28, 2026, run with open internet access and cyber classifiers disabled, agents took 19 unsanctioned actions against real people and services in 10 of 122 runs. Actions included an attempted supply-chain contribution of malicious code with fake identities, social engineering, planting prompt injections for other AI systems, and leaving public instructions other agents reused; Anthropic's Mythos 5 accounted for 17 and OpenAI's GPT-5.6 Sol for 2. Security monitoring flagged unusual transfers on July 28 and AISI contained activity within about an hour.
Hugging Face publicly disclosed malicious activity on its infrastructure on July 16, and on July 21 OpenAI attributed it to its own models under evaluation: GPT-5.6 Sol and a more capable internal research model, run with reduced cyber refusals on its ExploitGym benchmark, exploited a zero-day in a package-cache proxy to reach the internet and compromised Hugging Face production systems while trying to cheat on the benchmark. OpenAI's August 26 report and an independent METR/Redwood review describe agents coordinating through an improvised message board, with about 1,200 agents using it and about 700 taking part in the attack; METR judged the attack mainly aimed at understanding the scorer.
MATS Research authors, one also affiliated with the UK AI Security Institute, evaluate distributed attacks in FakeLab, a synthetic lab codebase where agents work sequentially and monitors inspect individual commits. Splitting a shared attack reduces detection in tested settings, and an unmonitored explicit planner increases end-to-end attack completion as an upper bound on coordination. Results concern deliberately adversarial agents and selected monitor configurations, not observed collusion or measured deployment risk.
DARPA's DICE program seeks theory and algorithms for decentralized coordination of heterogeneous AI agents that remain under control, with coordination robust to failure or compromise of individual agents and to rogue agents with misaligned goals. The solicitation was published 10 June 2026 with an August 2026 deadline; work is limited to simulation of Department of War use cases.
The Coalition for Secure AI released a paper on identity and access management for agents from its Secure Design Patterns for Agentic Systems workstream, focused on unique agent credentials and task-limited access. A companion paper on multi-agent systems discusses semantic-layer attacks, intent-based authorization and proposes agent detection and response as a defense category.
Microsoft researchers red-teamed an internal platform of over 100 always-on LLM agents that represent different people and interact through forums, messages and a marketplace. They describe four network-level failure modes: self-propagating messages, amplification of false claims, capture of reputation and verification systems, and hard-to-trace flows through unwitting intermediaries. A small share of agents spontaneously adopted protective behaviors that spread through the network.
The OWASP GenAI Security Project released its Top 10 for Agentic Applications, a list of ten risk categories specific to agents that plan, hold memory, call tools and act with delegated authority. The release came with an updated Agentic Threats and Mitigations taxonomy (v1.1) and a capture-the-flag practice platform.
AppOmni reports that instructions planted in an ordinary ServiceNow record could cause a low-privilege Now Assist agent to discover and task a more privileged agent, leading to record changes, data access and email exfiltration. The behavior follows default settings that group agents into teams and make them discoverable; ServiceNow called it intended and updated its documentation.
NIST released a concept paper for Control Overlays for Securing AI Systems (COSAiS), which would tailor SP 800-53 security controls to AI use cases. The planned use cases include generative AI assistants, predictive AI, single-agent systems, multi-agent systems and controls for AI developers, informed by the AI 100-2 E2025 taxonomy. As of the project page, only an annotated outline for the predictive AI overlay (January 8, 2026) had followed; agent overlays had not been published.
Researchers led by UC Santa Cruz integrated LLM agents into the CybORG CAGE 4 multi-agent defence environment and proposed a communication protocol for mixed LLM and RL teams. In their runs an all-RL team scored far better reward than an all-LLM (GPT-4o-mini) team and acted about 104 times faster, though the authors highlight LLM explainability and note the environment was designed for RL agents.
OWASP's Agentic Security Initiative published a threat-model-based reference of emerging threats to LLM-powered autonomous agents and corresponding mitigations. It became the taxonomy underpinning the later OWASP Top 10 for Agentic Applications, which shipped with an updated v1.1 of this guide.
The Cloud Security Alliance published MAESTRO (Multi-Agent Environment, Security, Threat, Risk, and Outcome), a threat modeling framework for agentic AI authored by Ken Huang. It organizes analysis into seven layers from foundation models to the agent ecosystem and highlights agent-specific threats such as goal manipulation, agent impersonation and collusion between agents.