Preference-optimization defense against prompt injection, later released as Meta SecAlign models.
Nasr, Carlini, Tramèr and 11 co-authors apply gradient, reinforcement learning, search and human red-teaming attacks to 12 published defenses. Most defenses originally reported near-zero attack success, but the adaptive attacks exceed 90% success against most, and human red-teamers succeeded on every challenge in the subset of defenses they were given.
Chen and colleagues (UC Berkeley and Meta) train models with preference optimization to prefer responses that follow the legitimate instruction over those that follow injected instructions. In the ACM CCS 2025 version they report injection success rates below 10% even for attacks more sophisticated than those seen in training, with utility similar to the undefended model; the October 2024 first version reported GCG-based injection success on Mistral-7B falling from 56% to 2%.