Controlled benchmarks, not deployed SOC performance. CyberSOCEval is multiple-choice rather than an agentic task; CyberClear has no analyst baseline and author-constructed references. APTInvestBench tests report-informed reconstructions under varying telemetry. ALERT-BENCH’s authors report that their AIDA architecture substantially improves replayed alert triage, but further scenarios remain within its dataset family, model performance varies widely, and analyst use and independently collected SOC telemetry are untested. Several benchmarks come from security vendors, including Microsoft and Simbian. CyberClear’s reported score comes from its authors’ own harness.
Qualified: Still standing, but later work narrows how far it applies.
Evidence
Best model reward 0.606 on log-investigation questions.
Multiple-choice questions over malware sandbox and threat reports, not agentic tasks; models far from saturating.
The best agent flagged 3.8% of malicious events in raw logs.
On a benchmark of reconstructing attack chains from defender logs, the best reported Strict score is 0.6477 out of 1, on the authors' own harness; the reference graphs were LLM-generated and LLM-verified with expert review, and there is no analyst baseline.
On controlled investigation reconstructions, acquired evidence exceeds formally cited support, and changing telemetry can change which actions remain supported.
The authors report that five baseline reasoning approaches miss at least 40.4% of attack alerts on their main telemetry replay; AIDA reduces this to 3.1% with higher escalation. Performance varies with model and scenario.
How it relates to other findings
Select a finding to see how it relates to others. Arrows point from the newer finding to the one it supports, qualifies, contests, or supersedes.
- This finding qualifies Controlled trials run by Microsoft report that its security assistants make analysts faster and more accurate.Gains appear in assisted work; baseline autonomous investigation and triage approaches can remain weak in controlled security-log benchmarks.
Key questions that rely on this finding
- Can AI agents defend and oversee systems on their own?Not reliably. Some designs improve benchmark triage, but autonomous defense lacks operational validation and production oversight of agents has documented gaps.
Status history
- 2025-07-14ReportedExCyTIn-Bench results. · record
- 2025-09-24CorroboratedCyberSOCEval finds similar limits. · record
- 2026-09-25CorroboratedcorrectionCyberSOCEval tests multiple-choice question answering, not agents on investigation or hunting. Corroboration rests on Simbian's Cyber Defense Benchmark, where the best of five models flagged 3.8% of malicious events in raw logs. · record
- 2026-10-13QualifiedALERT-BENCH identifies large baseline triage failures but reports substantial improvement from AIDA’s separate challenge and adjudication. The weak-performance claim is narrowed to baseline approaches; operational reliability remains unestablished. · record