Topics/Defense

SOC automation

Triage, detection engineering, and analyst assistance.

16 records4 findings1 openings7 benchmarks and toolsLatest record
RangeLanes
14 of 16 records in view

Use the arrow keys to move between records, Home and End to jump to the first and last, and Enter to select one.

Agents find real bugsAgents in real operationsGated capability, incidents in the labAttackCapabilityDefensePolicyJan 25Jul 25Jan 26Jul 26
Full record · drag to choose a range
202420252026

Select a mark to read the record. Mark size shows editorial significance. Hollow marks are dated to the month. Era bands are editorial labels.

Records in view

14 records · newest first
Oct 2026
Oct 7, 2026
ALERT-BENCH finds triage failures; AIDA reduces missed attacks in telemetry replay
DefenseBenchmarkVirginia Tech, North Carolina State University, Google

Researchers introduce ALERT-BENCH, requiring agents to investigate detector-generated alerts through a live SIEM over replayed public telemetry. On the main 1,247-alert sample, five reasoning approaches each miss at least 40.4% of attack-related alerts, while the authors’ AIDA architecture reduces the false-negative rate to 3.1% under the same GPT-4.1 configuration, with more alerts escalated to analysts. Additional scenario and model tests remain within the studied dataset family; analyst use and independent SOC validation are untested.

Sep 2026
Sep 30, 2026
APTInvestBench finds autonomous investigators lose citation support when telemetry changes
DefenseBenchmarkZhongguancun Laboratory

Zhongguancun Laboratory researchers introduce APTInvestBench, built from report-informed attack reconstructions under varied log-collection conditions. In their ten-scenario comparison of eleven models, agents acquire sufficient evidence for 44.3% of recoverable attack actions on average, but their formal citations support 25.0%; similar aggregate citation coverage can conceal losses of support for actions that remain recoverable. These are controlled benchmark investigations, not measurements of deployed SOC performance.

Sep 26, 2026
CyberClear benchmarks LLM agents on reconstructing APT attack chains from long defender logs
DefenseBenchmarkHong Kong Polytechnic University, OpenClaw, Southeast University

Chen and colleagues (arXiv v1, under review for ICLR 2027) introduce CyberClear, 450 instances built from public APT log datasets in which an agent must turn long defender logs, without prior attack clues, into a provenance graph with ATT&CK-mapped steps. References were generated by an LLM and passed automatic verification and expert review; scoring compares graph code with five LLM-judge dimensions. They also propose CyberProvenance, a multi-agent harness that reproduces predicted attack steps in isolated lab environments and refines the graph, and report it best on most semantic metrics while the highest Strict score remains 0.6477 out of 1.

Sep 21, 2026
UK NCSC blog argues defenders cannot use agentic AI as attackers do and proposes a five-dimension action risk framework
PolicyGuidanceUK National Cyber Security Centre, CETaS (Alan Turing Institute)

Dave Chismon, the NCSC's CTO for Architecture, argues that offensive problems are mostly technical with clear success states while defensive problems are mostly organizational, so defenders cannot put AI to work as attackers can and agentic defense may struggle to keep pace with AI-enabled attacks. He proposes rating a defensive action's riskiness on five dimensions (potency, scope, criticality, rollout confidence, recoverability) and starting with low-potency tasks where AI advises a human. The post also says the NCSC and DCMS are working on a national-scale agentic cyber defence ecosystem called Cyber Shield and that the NCSC seeks research on proving low-risk actions are low risk.

Sep 5, 2026
Year-long SOC fieldwork finds analysts reused an agentic AI companion's output in over 90% of tickets
DefensePaperUniversity of South Florida

University of South Florida researchers embedded in a working SOC for over a year built and deployed an LLM-based agentic companion to handle high-volume, low-priority tickets, with analysts using it in the final four months. They report that companion outputs were reused in analysts' closing reports in more than 90% of cases, and that analysts who shaped the companion's behaviour came to trust it more.

Jul 2026
Jul 16, 2026
Cost-aware evaluation finds defensive SOC agents do not scale with compute like offensive CTF agents
DefensePaper

Researchers evaluate security agents at fixed cost levels on offensive Cybench challenges and defensive Splunk BOTS v1 investigations, splitting spend into inference and tool use. They find offensive success rises with test-time compute, while defensive investigation depends more on disciplined tool use and telemetry navigation, and argue benchmarks should report cost and operational fit alongside success.

Apr 2026
Apr 21, 2026
Threat-hunting benchmark finds best LLM agent flags only 3.8% of malicious events in raw logs
DefenseBenchmarkSimbian AI

A technical report from security vendor Simbian AI presents the Cyber Defense Benchmark, which asks agents to hunt through 75,000-135,000 raw Windows event log records per episode, with no guiding questions, and flag the timestamps of malicious events drawn from 106 OTRF attack procedures. In the first version, the best of five frontier models (Claude Opus 4.6) flagged only 3.8% of malicious events on average and no model met the authors' bar of 50% recall on every ATT&CK tactic. A revision two days later, with more models and a new coverage metric, reached the same no-pass conclusion.

Mar 2026
Mar 13, 2026
Microsoft's CTI-REALM benchmark tests agents turning threat intel into validated detection rules
DefenseBenchmarkMicrosoft

CTI-REALM places agents in a tool-rich environment where they read threat intelligence reports, explore telemetry, iterate KQL queries and produce Sigma and KQL detection rules across Linux, AKS and Azure cloud scenarios. The paper's evaluation of 16 model configurations found Claude Opus 4.6 (High) best at 0.637, with cloud detection hardest; Microsoft's blog later added an early Claude Mythos Preview snapshot scoring 0.685.

Nov 2025
Nov 17, 2025
Microsoft RCT finds phishing triage agent raised analysts' true positives per minute up to 6.5x
DefensePaperMicrosoft

Microsoft reports a randomized controlled trial of its own Security Copilot Phishing Triage Agent. In the trial, 167 external security analysts each triaged a 25-email queue drawn from a curated corpus of emails reported by Microsoft employees. In the scenario where the agent classified every corpus email correctly, analysts with the agent found 6.5 times as many true positives per minute as the control group and scored 77% higher on F1; with the agent's accuracy set to 80% and a 20% malicious rate, the productivity gain fell to 3.1 times. Analysts with the agent spent 53% more time on malicious emails and did not simply confirm its malicious verdicts, but they were more likely to accept its benign verdicts, including planted false negatives.

Sep 2025
Aug 2025
Aug 5, 2025
Microsoft's Project Ire agent autonomously reverse engineers and classifies malware
DefenseTool releaseMicrosoft

Microsoft Research describes Project Ire, a prototype LLM agent that uses decompilers and binary analysis tools to reverse engineer software and classify it as malicious or benign, producing an auditable chain-of-evidence report. Microsoft reports 0.98 precision and 0.83 recall on a Windows driver dataset, but 0.26 recall on about 4,000 hard real-world files, and plans to deploy it in Defender as Binary Analyzer.

Jul 2025
Jul 14, 2025
Microsoft's ExCyTIn-Bench evaluates LLM agents on multi-step threat investigation over Sentinel logs
DefenseBenchmarkMicrosoft

ExCyTIn-Bench builds threat-investigation questions from graphs of security logs collected in a controlled Azure tenant with simulated multi-step attacks, and asks agents to query the logs to answer them. In the July 2025 version the best model (o4-mini) reached a reward of 0.368; in the May 2026 revision, accepted at ICML 2026, the best (Claude Opus 4.5) reached 0.606, which the authors say leaves substantial headroom.

Apr 2025
Apr 4, 2025
Google announces Sec-Gemini v1, an experimental model for security operations workflows
DefenseTool releaseGoogle

Google announced Sec-Gemini v1, an experimental model combining Gemini with Google Threat Intelligence, OSV and Mandiant data for tasks such as incident root-cause analysis and vulnerability impact assessment. Google reports it outperforms other models by at least 11% on CTI-MCQ and 10.5% on CTI-Root Cause Mapping, and offered free research access to selected organizations.

Mar 2025
Mar 24, 2025
Microsoft announces Security Copilot agents for phishing triage, alert triage and remediation
DefenseTool releaseMicrosoft, OneTrust, Aviatrix

Microsoft announced Microsoft-built Security Copilot agents, including a Phishing Triage Agent in Defender, alert triage agents in Purview, a Conditional Access Optimization Agent, a Vulnerability Remediation Agent in Intune and a Threat Intelligence Briefing Agent, plus five partner agents. Preview was planned from April 2025; the announcement contains no evaluation of agent accuracy.

Findings

Research openings

Benchmarks and tools