Anthropic's Frontier Red Team reports simplified experiments on whether small open-weight models could replicate by running themselves on low-powered, poorly defended internet-connected devices. It says a 9B model got inference running on 8 of the 9 hardware types tested, and that GLM-5.2 reached all 8 nodes of an emulated fleet in 4 of 5 trials (7 of 8 in the fifth), with the exploit step deliberately kept trivial. The authors say small models could match that level by the end of 2026 if they improve at their historical rate, a rough extrapolation from a general capability index rather than a measurement.
CyberSecEval
Meta's suite of cybersecurity evaluations for language models, part of Purple Llama.
Records citing CyberSecEval
OpenAI describes IH-Challenge, a reinforcement learning dataset of simple, programmatically graded conflicts between higher- and lower-privilege instructions designed to avoid shortcuts such as over-refusal. A GPT-5 Mini variant trained on it (GPT-5 Mini-R) improved on instruction-hierarchy benchmarks and on CyberSecEval 2 and an internal prompt injection benchmark, with little capability loss; the dataset is publicly released.
CyberSOCEval adds two open-source SOC benchmarks to CyberSecEval 4: malware analysis questions built from sandbox detonation reports, and threat intelligence reasoning over unstructured reports. The authors find larger, newer models do better, reasoning models gain less than in coding and math, and current models are far from saturating the tasks.
Meta introduced AutoPatchBench, part of CyberSecEval 4, with 136 fuzzing-identified C/C++ vulnerabilities and verified fixes, plus a 113-case Lite subset with single-function root causes. Patches are checked by build and crash reproduction, then fuzzing and white-box differential testing; Meta's reference agent generated crash-stopping patches in about 60% of cases, but only 5-11% passed the stricter checks.