Findings/soc-agents-weak-on-realistic-benchmarks

Baseline agent approaches can fall well short of reliable investigation and triage on security-log benchmarks.

Qualifiedmeasured6 evidence records from 6 independent sourcesassistant-drafted
Scope: what this does not show

Controlled benchmarks, not deployed SOC performance. CyberSOCEval is multiple-choice rather than an agentic task; CyberClear has no analyst baseline and author-constructed references. APTInvestBench tests report-informed reconstructions under varying telemetry. ALERT-BENCH’s authors report that their AIDA architecture substantially improves replayed alert triage, but further scenarios remain within its dataset family, model performance varies widely, and analyst use and independently collected SOC telemetry are untested. Several benchmarks come from security vendors, including Microsoft and Simbian. CyberClear’s reported score comes from its authors’ own harness.

Qualified: Still standing, but later work narrows how far it applies.

Evidence

How it relates to other findings

supportsqualifiescontestssupersedes
ReportedCorroboratedQualifiedContestedSupersededRevalidate· node size = evidence records · columns group by topic

Select a finding to see how it relates to others. Arrows point from the newer finding to the one it supports, qualifies, contests, or supersedes.

Key questions that rely on this finding

Status history

  1. 2025-07-14ReportedExCyTIn-Bench results. · record
  2. 2025-09-24CorroboratedCyberSOCEval finds similar limits. · record
  3. 2026-09-25CorroboratedcorrectionCyberSOCEval tests multiple-choice question answering, not agents on investigation or hunting. Corroboration rests on Simbian's Cyber Defense Benchmark, where the best of five models flagged 3.8% of malicious events in raw logs. · record
  4. 2026-10-13QualifiedALERT-BENCH identifies large baseline triage failures but reports substantial improvement from AIDA’s separate challenge and adjudication. The weak-performance claim is narrowed to baseline approaches; operational reliability remains unestablished. · record