Meta

Meta disclosed that a misconfiguration by its testing vendor Irregular let one of its models reach the internet during evaluation, where it exploited a vulnerability in a third-party service. Meta did not name the model; CBS News, relaying Reuters, reports sources told The Information it was Muse Spark 1.1. Meta said it learned of the incident from Irregular and would publish a retrospective.

Meta proposes that, within a session, an agent should have at most two of three properties: processing untrustworthy inputs, accessing sensitive systems or private data, and changing state or communicating externally. If all three are needed, the agent should not act autonomously and needs human approval or other validation. Meta illustrates this with travel, research and internal coding agent examples.

CyberSOCEval adds two open-source SOC benchmarks to CyberSecEval 4: malware analysis questions built from sandbox detonation reports, and threat intelligence reasoning over unstructured reports. The authors find larger, newer models do better, reasoning models gain less than in coding and math, and current models are far from saturating the tasks.

Meta open-sources LlamaFirewall, combining PromptGuard 2 (a jailbreak and injection detector), AlignmentCheck (a chain-of-thought auditor for goal hijacking) and CodeShield (static analysis of generated code). On AgentDojo, Meta reports that the combination cut attack success from 17.63% to 1.75% while utility fell from 47.73% to 42.68%.

Meta introduced AutoPatchBench, part of CyberSecEval 4, with 136 fuzzing-identified C/C++ vulnerabilities and verified fixes, plus a 113-case Lite subset with single-function root causes. Patches are checked by build and crash reproduction, then fuzzing and white-box differential testing; Meta's reference agent generated crash-stopping patches in about 60% of cases, but only 5-11% passed the stricter checks.

Chen and colleagues (UC Berkeley and Meta) train models with preference optimization to prefer responses that follow the legitimate instruction over those that follow injected instructions. In the ACM CCS 2025 version they report injection success rates below 10% even for attacks more sophisticated than those seen in training, with utility similar to the undefended model; the October 2024 first version reported GCG-based injection success on Mistral-7B falling from 56% to 2%.