Agentic Cyber Explorerby Fide AI · PreviewPreview
Chronicle/Defense & research

ALERT-BENCH finds triage failures; AIDA reduces missed attacks in telemetry replay

DefenseBenchmarkSignificance assistant-drafted

Researchers introduce ALERT-BENCH, requiring agents to investigate detector-generated alerts through a live SIEM over replayed public telemetry. On the main 1,247-alert sample, five reasoning approaches each miss at least 40.4% of attack-related alerts, while the authors’ AIDA architecture reduces the false-negative rate to 3.1% under the same GPT-4.1 configuration, with more alerts escalated to analysts. Additional scenario and model tests remain within the studied dataset family; analyst use and independent SOC validation are untested.

Why it matters

It connects triage decisions to evidence retrieval and independent challenge, showing both improved attack coverage and the resulting analyst workload.

Key facts

As stated in the sources, with where to find them.

  • The main comparison uses 1,247 alerts, including 225 attack-related and 1,022 benign alerts, from a four-day AIT-LDSv2 scenario. Each approach uses GPT-4.1 and the same evidence interface; investigators begin from the alert rather than a preselected evidence packet.Sections III–IV; Table I
  • AIDA records a proposed decision before separate challenge and adjudication. On the main sample its reported F1 is 0.958 versus 0.371–0.744 for the five baselines; false-negative rate falls to 3.1% from the best baseline’s 40.4%, while the share escalated rises to 18.4% from baseline rates of 8.5–14.0%.Abstract; Section VI; Figure 6
  • The authors also test five open-weight models on a 300-alert subset and evaluate a separate five-day scenario containing 337 alerts with 5.93% attack prevalence. These provide broader tests within the same dataset family, not evaluation on independently collected operational SOC data. On the 300-alert subset Qwen3.5 9B reaches F1 0.952, while the other tested open-weight models range from 0.098 to 0.777, showing substantial backend dependence.Section VI-D; Discussion and Limitations
  • The evaluation begins after a detector raises an alert and depends on the accessible investigation context. Its experiments do not measure analysts’ review effort, override decisions or effectiveness in operational deployment.Discussion and Limitations

Findings that cite this record

Key questions this bears on

Sources

Related records