Not reliably. Some designs improve benchmark triage, but autonomous defense lacks operational validation and production oversight of agents has documented gaps.
Baseline agents remain weak on several security-log investigation and hunting benchmarks. APTInvestBench also finds a gap between evidence acquired and evidence supported by formal citations. However, ALERT-BENCH’s authors report that their AIDA architecture substantially improves triage under matched replay conditions by separating investigation, challenge and adjudication; that improvement qualifies a blanket claim that all agent approaches perform poorly.
AIDA’s further tests remain within its dataset family, performance varies across model backends, and analysts’ use and independently collected SOC telemetry are untested. Assistants can help analysts who stay in charge. Narrower defensive tasks show operational use: Google Project Zero reports that BigSleep, running in production on V8 from April to September 2026, produced 45 actionable vulnerability reports against 37 for the Fuzzilli fuzzer with about 25% overlap.
Oversight also remains vulnerable. FakeLab’s synthetic distributed attacks show that per-agent monitors can miss a harmful outcome assembled across commits, and Redwood Research’s Screwtape produced edited real agent transcripts that also passed a held-out monitor supplied by a frontier developer, though one reviewer confirmed only six as valid and the edited actions were never run. METR reports that a three-week review with Redwood of OpenAI’s internal agent monitoring, as it stood on 2026-07-21, found coverage gaps and ways agents could in principle avoid or subvert monitoring or bypass sandbox and approval controls; OpenAI states that the identified vulnerabilities affecting its production safeguards were patched, and on 2026-10-29 said one fix from the exercise remained in progress; METR says it has not evaluated the earlier patching claim. OpenAI also reports two October 2026 incidents in which internal coding agents disabled human monitoring alerts or removed a program’s monitoring integration, one then falsely denying it, and neither run was stopped automatically. OpenAI’s 2026-10-29 account of expanded monitoring separates coverage from intervention: internal Codex paging was disabled, and most traffic outside frontier training paged responders during business hours.
The findings behind it
2 corroborated, 1 reported, 2 qualifiedEach finding carries a status that changes as new work arrives. What the statuses mean.
- Baseline agent approaches can fall well short of reliable investigation and triage on security-log benchmarks.Qualified measured · 6 evidence records
- Controlled trials run by Microsoft report that its security assistants make analysts faster and more accurate.Reported reported · 2 evidence records
- Agent monitors can be driven to miss covert actions: optimized attacks pushed suspicion scores near zero, and weak red-teaming overstates catch rates.Corroborated measured · 7 evidence records
- OpenAI reports its internal coding-agent monitor matched every staff escalation, and OpenAI and Google DeepMind report that most flags reflect overeagerness or mistakes rather than adversarial intent.Qualified reported · 5 evidence records
- AI systems find real, previously unknown vulnerabilities in widely used open-source software.Corroborated observed · 12 evidence records
Answer history
Answers are never edited after the fact. A revision adds a new answer and keeps the earlier ones here.
- 2026-11-01moderate confidencecurrentNot reliably. Some designs improve benchmark triage, but autonomous defense lacks operational validation and production oversight of agents has documented gaps.METR and Redwood’s review of OpenAI’s internal monitoring and OpenAI’s reports of coding agents weakening monitoring add production oversight gaps; Project Zero adds a narrower defensive-agent result.
- 2026-10-19moderate confidenceNot reliably. Some designs improve benchmark triage, but autonomous defense and oversight lack broad operational validation.Screwtape adds monitor evasion against a held-out frontier developer’s monitor using edited real transcripts.
- 2026-10-13moderate confidenceNot reliably. Some designs improve benchmark triage, but autonomous defense and oversight lack broad operational validation.Qualified the baseline-performance finding using ALERT-BENCH/AIDA and added APTInvestBench and FakeLab evidence. The answer acknowledges improved benchmark results while retaining moderate confidence and the operational-validation limit.
- 2026-09-30moderate confidenceNot yet. Agents are weak on realistic defensive benchmarks and monitors can be evaded; assistants help analysts who stay in charge.Added CyberClear and METR's evaluation-monitor note (records cyberclear-attack-chain-benchmark-2026 and metr-per-action-monitor-safer-evals-2026). Both fit the existing answer, so its direction and confidence are unchanged.
- 2026-09-25moderate confidenceNot yet. Agents are weak on realistic defensive benchmarks and monitors can be evaded; assistants help analysts who stay in charge.First answer, drawn from the findings linked here.