Methods/Defense

Human approval gates

Requiring a person to approve consequential agent actions before they happen.

20 records6 attack11 defense3 policy4 findings (2 measured)First recorded 2025-07assistant-drafted

How it works

The agent pauses and shows the proposed action. Effectiveness depends on what the approval screen shows and how many approvals people are asked for.

Known limits

Approval fatigue and incomplete approval displays weaken the control.

What we know

3 corroborated, 1 revalidate

Records over time

RangeLanes
20 of 20 records in view

Use the arrow keys to move between records, Home and End to jump to the first and last, and Enter to select one.

Agents find real bugsAgents in real operationsGated capability, incidents in the labAttackCapabilityDefensePolicyJan 25Jul 25Jan 26Jul 26
Full record · drag to choose a range
2026

Select a mark to read the record. Mark size shows editorial significance. Hollow marks are dated to the month. Era bands are editorial labels.

Records in view

20 records · newest first
Nov 2026
Nov 10, 2026
AlphaProof Code checks specified Signal security properties at machine-code level
DefensePaperGoogle DeepMind, Google

Google DeepMind describes AlphaProof Code, combining human-approved specifications with agent-generated proofs checked by Lean. Its libsignal study proves specified properties of PQXDH and the Triple Ratchet at the AArch64 machine-code level. Assurance depends on the specifications, formal semantics and other stated trust assumptions; it does not cover every property of the Signal application.

Nov 2, 2026
Trail of Bits reports Codex-assisted backports delivered for 149 Linux CVEs
DefenseEvaluation reportTrail of Bits, OpenAI

Trail of Bits reports that its Codex workflow helped deliver fixes for 149 distinct CVEs into stable Linux kernel releases by October 29, with additional package work for Fedora and Ubuntu. Agents prepare backports and evidence; engineers review and submit them to maintainers. Reported human-time savings exclude setup, regression testing and maintainer acceptance.

Oct 2026
Oct 19, 2026
Microsoft extends Project Perception to treat AI agents as insiders, with agent identities, task-scoped permissions and risk-based access
DefenseTool releaseMicrosoft

Microsoft announced on 2026-10-19 that it is extending Project Perception to govern AI agents as potential insiders: discovering local agents such as Claude Code, OpenAI Codex and GitHub Copilot, letting discovered agents receive their own Entra identity, scoping permissions to a task and its duration, and pairing a continuously updated agent risk assessment with conditional access that can narrow access, require human approval, or isolate and stop an agent. Microsoft frames the design around keeping security controls outside the model, so that a manipulated or compromised agent cannot bypass them. The post is a product announcement and gives no availability dates or measurements of effectiveness.

Oct 1, 2026
Salt Labs reports a resolved Manus flaw that ran injected code before warning
AttackVulnerability disclosureSalt Security, Manus

Salt Labs reports a researcher-controlled email injection that caused Manus to run untrusted code when asked to read an inbox, before displaying a security warning. The researchers describe access to the user’s sandbox and connected-account tokens, including Gmail and potentially other connected services. Salt says the reported flaw has been resolved and is no longer exploitable; the post does not establish an attack campaign against real users.

Sep 2026
Sep 30, 2026
TrustProbe reports 104 skill-mediated trust failures across eleven agents under permissive test settings
AttackPaperInstitute of Information Engineering, Chinese Academy of Sciences, Worcester Polytechnic Institute, University of Chinese Academy of Sciences

Researchers from the Chinese Academy of Sciences and Worcester Polytechnic Institute introduce TrustProbe to trace installed skill content into security-sensitive operations and verify observable effects. Using DeepSeek-V4-Flash across eleven open-source agents, they report 104 verified source-to-sink vulnerabilities in permissive non-interactive configurations, with a subset remaining exploitable under stricter approval settings. Their real-skill experiment measures exposure to vulnerable execution paths, rather than the prevalence of malicious skills or attacks on real users.

Sep 27, 2026
METR reports a live per-action blocking monitor for its own evals, with validation results and a list of gaps in its evidence
DefenseEvaluation reportMETR, UK AI Security Institute

In a research note, METR describes a monitor it deployed on its own agent evaluations after the OpenAI, Anthropic and UK AISI incidents: an LLM judge scores each tool call before it runs through Inspect's approval chain, and calls scoring 3/10 or higher are held for human review, with the run halted. METR reports that a UK AISI replay flagged all 10 malicious transcripts from that incident (minimum score 8/10) with about 98% of 25,000 benign actions scoring 0/10, that 13 of 13 malicious actions in synthetic transcripts were flagged, and that 5 of 20,000 sampled benign actions (0.025%) were flagged. METR frames the note as an argument with self-graded evidence, and lists gaps it found, including a researcher running evals without the monitor and a hand-built spoofed user turn that lowered suspicion scores.

Jul 2026
Jul 23, 2026
UK AISI Control Red Team finds evasion weaknesses in Google DeepMind and Anthropic agent monitors
DefenseEvaluation reportUK AI Security Institute, Google DeepMind, Anthropic

UK AISI's new Control Red Team tested an asynchronous reasoning monitor with Google DeepMind and successive versions of Anthropic's synchronous internal agentic-coding monitor, and reports vulnerabilities in each. Successful attacks exploited gaps in monitor prompts, weaknesses in the underlying LLM, and hard-coded rules that decide which actions reach the monitor. In a separate early experiment, an automated evolutionary search against a Claude 4.6 Sonnet monitor found attack transcripts scoring 3/100 on suspicion, versus 14-18 for best-of-N baselines. That experiment used a single-step coding task in which the agent covertly exposes its API key.

Jun 2026
Jun 3, 2026
Frontier Model Forum issue brief catalogs emerging security practices for AI agents
PolicyGuidanceFrontier Model Forum

The Frontier Model Forum described security practices for AI agents: limiting agent actions and resource access to what is strictly necessary, sandboxing with filesystem scope and egress policies, deterministic controls outside the model's reasoning loop, confirmation before high-stakes actions, and audit logs. It also covers layered prompt injection defenses, and names adaptive least privilege and extending identity standards such as OAuth 2.0 to agents as promising or developing areas.

May 2026
May 15, 2026
UK NCSC advises incremental agentic AI adoption with minimal, expiring permissions
PolicyGuidanceUK National Cyber Security Centre

NCSC authors advise deploying agentic AI incrementally through tightly bounded pilots, granting agents only the minimum permissions with temporary credentials, and defining in advance who approves access, monitors behavior and can halt the agent. They recommend incident response plans for agent failure and loss-of-control scenarios.

May 10, 2026
MonitoringBench shows refined covert attacks cut an Opus 4.5 monitor's catch rate from 95% to 60%
DefenseBenchmarkRedwood Research

Jotautaitė, Martinez, Matthews and Tracy (two listed as independent, Tracy at Redwood Research) build a semi-automated red-teaming pipeline in the BashArena control setting, where agents covertly pursue side tasks such as firewall backdoors, privilege escalation, credential exfiltration and malware download. MonitoringBench's 2,644 attack trajectories lowered a Claude Opus 4.5 monitor's catch rate at 1% false positives from 94.9% on directly elicited attacks to 60.3% on refined ones.

Mar 2026
Mar 19, 2026
OpenAI reports monitoring tens of millions of internal coding agent trajectories for misalignment
DefenseEvaluation reportOpenAI

OpenAI describes an asynchronous monitor powered by GPT-5.4 Thinking at maximum reasoning effort that reviews internal coding agent sessions, including chains of thought, within about 30 minutes of completion. OpenAI reports that over five months it covered tens of millions of trajectories, with no real conversation reaching the highest severity and about 1,000 moderate-severity alerts, many from deliberate internal red-teaming, and that it flagged every interaction employees had independently reported. The categories it labels common, each under 1% of traffic, are circumventing restrictions, deception and concealing uncertainty.

Feb 2026
Feb 25, 2026
Check Point: Claude Code project files could run commands and leak API keys before trust prompt
AttackVulnerability disclosureCheck Point Research, Anthropic

Check Point Research found that a cloned repository's Claude Code configuration could run hooks, start MCP servers before the user approved them, and redirect API traffic so the user's Anthropic API key was sent to an attacker (CVE-2025-59536, CVE-2026-21852). Anthropic fixed the issues between August and December 2025 by deferring execution and API calls until after the trust dialog.

Feb 20, 2026
Anthropic releases Claude Code Security in limited preview to scan code and propose patches
DefenseTool releaseAnthropic

Anthropic released Claude Code Security as a limited research preview for Enterprise and Team customers, with expedited free access for open-source maintainers. The tool reasons about data flow across a codebase, re-examines each finding in a multi-stage verification pass, assigns severity and confidence ratings, and proposes patches that are applied only with human approval.

Nov 2025
Nov 7, 2025
OpenAI describes its layered approach to prompt injection as a frontier security challenge
DefenseFrameworkOpenAI

OpenAI describes prompt injection as social engineering aimed at AI agents and lists its layered defenses: instruction-hierarchy safety training, automated red-teaming, AI-based monitors that can be updated quickly, sandboxing of code-running tools, link approval, confirmation before sensitive steps, logged-out mode in Atlas, and a watch mode on sensitive sites that pauses the agent if the user leaves the tab. It cites thousands of hours of prompt-injection-focused red teaming and a bug bounty, and says it has not yet seen significant attacker adoption of the technique.

Oct 2025
Oct 31, 2025
Meta proposes the 'Agents Rule of Two' for limiting prompt injection impact
DefenseFrameworkMeta

Meta proposes that, within a session, an agent should have at most two of three properties: processing untrustworthy inputs, accessing sensitive systems or private data, and changing state or communicating externally. If all three are needed, the agent should not act autonomously and needs human approval or other validation. Meta illustrates this with travel, research and internal coding agent examples.

Oct 20, 2025
Anthropic adds OS-level filesystem and network sandboxing to Claude Code and open-sources the runtime
DefenseTool releaseAnthropic

Anthropic describes sandboxing for Claude Code that restricts file writes to permitted directories and routes network traffic through a proxy that only allows approved hosts, so a prompt-injected agent cannot modify sensitive files or exfiltrate data freely. Anthropic says internal use showed an 84% reduction in permission prompts, and it released the sandbox runtime, built on bubblewrap and macOS seatbelt, as an open-source research preview.

Aug 2025
Aug 12, 2025
GitHub Copilot agent could be prompt-injected into disabling its own approvals (CVE-2025-53773)
AttackVulnerability disclosureJohann Rehberger (Embrace The Red), Microsoft, GitHub

Johann Rehberger showed that injected instructions in project content could make GitHub Copilot in VS Code edit workspace settings to switch off command confirmations, after which it could run arbitrary terminal commands. He reported it on 2025-06-29 and Microsoft patched it in the August 2025 Patch Tuesday.

Aug 6, 2025
SafeBreach shows Google Calendar invites can hijack Gemini for Workspace agents
AttackVulnerability disclosureSafeBreach, Tel Aviv University, Technion

SafeBreach researchers showed that instructions in calendar invite titles or email subjects, retrieved when a user asked Gemini about their schedule, could trigger tool misuse such as deleting events, opening URLs, exfiltrating email subjects, and controlling connected smart-home devices. They disclosed to Google in February 2025, and Google deployed layered mitigations including confirmations, URL handling policies and injection classifiers.

Aug 5, 2025
MCPoison: Cursor trusted approved MCP configs even after their commands changed (CVE-2025-54136)
AttackVulnerability disclosureCheck Point Research, Cursor

Check Point Research found that Cursor bound MCP approval to a configuration's name rather than its contents, so a collaborator with repository write access could swap an approved harmless command for a malicious one that ran on each project open. Cursor 1.3, released 2025-07-29, prompts for approval on any MCP configuration change.

Jul 2025
Jul 16, 2025
Coalition for Secure AI publishes Principles for Secure-by-Design Agentic Systems
PolicyFrameworkCoalition for Secure AI, OASIS Open

The Coalition for Secure AI, an OASIS Open Project, published three principles for agentic systems. The principles call for agents that are human-governed and accountable, bounded and resilient with strict, purpose-specific entitlements, and transparent and verifiable through secure AI supply chain controls and telemetry that supports monitoring and forensics.

All records