Apr 21, 2026
Threat-hunting benchmark finds best LLM agent flags only 3.8% of malicious events in raw logs
A technical report from security vendor Simbian AI presents the Cyber Defense Benchmark, which asks agents to hunt through 75,000-135,000 raw Windows event log records per episode, with no guiding questions, and flag the timestamps of malicious events drawn from 106 OTRF attack procedures. In the first version, the best of five frontier models (Claude Opus 4.6) flagged only 3.8% of malicious events on average and no model met the authors' bar of 50% recall on every ATT&CK tactic. A revision two days later, with more models and a new coverage metric, reached the same no-pass conclusion.