Zhongguancun Laboratory researchers introduce APTInvestBench, built from report-informed attack reconstructions under varied log-collection conditions. In their ten-scenario comparison of eleven models, agents acquire sufficient evidence for 44.3% of recoverable attack actions on average, but their formal citations support 25.0%; similar aggregate citation coverage can conceal losses of support for actions that remain recoverable. These are controlled benchmark investigations, not measurements of deployed SOC performance.
Why it matters
It distinguishes missing telemetry from an agent failing to find evidence or preserve it in its report, making aggregate investigation scores easier to interpret.
Key facts
As stated in the sources, with where to find them.
- The resource contains 370 cases across seven SOC-inspired telemetry conditions, derived from 56 report-informed attack reconstructions and approximately 16.4 million log records. The main evaluation concentrates on ten of the 56 scenarios.v1, Abstract; Section 5, Experimental setup
- Eleven-model comparisons use Codex; framework comparisons hold MiniMax-M3 fixed across Codex, Claude Code, OpenCode and Ref-ReAct. Across the seven telemetry conditions, models acquire sufficient evidence for 44.3% of recoverable actions on average, while formal citations support 25.0%.v1, Sections 5 and 5.1; Figure 3
- On actions recoverable in both Full and endpoint-only telemetry, pooled citation coverage changes from 27.5% to 25.9%. The authors report that 64.5% of actions supported in Full reports remain supported in endpoint-only reports; newly covered actions partly offset lost support. These comparisons allow the available supporting records to differ.v1, Section 5.2; Figure 4
- A stricter comparison holds registered candidate records and sufficient-evidence sets unchanged. Investigation results still differ across telemetry views; all four evaluated frameworks have lower citation coverage under endpoint-only telemetry.v1, Section 5.3; Figure 6
Findings that cite this record
Key questions this bears on
- Can AI agents defend and oversee systems on their own?Not reliably. Some designs improve benchmark triage, but autonomous defense lacks operational validation and production oversight of agents has documented gaps.
Sources
Related records
Sep 26, 2026
Oct 27, 2026
Oct 13, 2026
Oct 12, 2026
Sep 28, 2026
Oct 7, 2026