Organizations/academic

Carnegie Mellon University

3 records2 capability1 defense
Oct 22, 2026
AgentPhishing tests email-agent decisions against adaptive synthetic phishing
DefenseBenchmarkCarnegie Mellon University

Carnegie Mellon researchers introduce AgentPhishing, separating phishing detection from compliance with malicious requests. In their universal-skill evaluation, four of nine additional target agents had detection below 10% and compliance above 45%. The benchmark scores selected actions and drafted replies, not externally executed emails or completed real-world harm.

Oct 13, 2026
CyberArena benchmark pits 12 frontier agents against each other in live attack-and-defense matches over 466 games
CapabilityBenchmarkUC Berkeley, UC Santa Cruz, Purdue University

A team from 2077AI and several universities presents CyberArena, in which two model-harness entrants host identical vulnerable services in isolated sandboxes and, over three 20-minute rounds, try to capture rotating flags from the opponent while patching their own service under health checks. They report standings for 12 agents over 466 valid games on 40 challenges, with GPT-5.6 Sol at an expected score of 78.6 and Claude Opus 5 at 73.8, and say contested play lowers capture rates relative to solo play. An LLM-run audit of all trajectories reports recurring defender errors such as partial patches and repair claims resting on stale evidence; the authors note it is not human verification of every finding.

May 13, 2026
ExploitBench grades AI exploit development as a 16-step capability ladder on V8 bugs
CapabilityBenchmarkCarnegie Mellon University, Bugcrowd

Carnegie Mellon researchers released ExploitBench, which scores exploitation progress on 41 V8 JavaScript-engine vulnerabilities across 16 flags from reaching the bug through arbitrary read/write, control-flow hijack and code execution. The paper reports that public models routinely reach and crash vulnerable code but rarely achieve arbitrary code execution, while one private frontier model succeeded on roughly half of cases.