A Google DeepMind preprint tests collaborative attacker agents on a synthetic instruction-hierarchy benchmark with randomly selected, non-malicious tool-call goals. Stronger attackers and larger query budgets expose failures that static tests miss. The experiments isolate model behavior and do not measure successful real-world data theft or a complete deployed defense stack.
Why it matters
It shows why robustness claims depend on adaptive attacker resources and on which safeguards are actually included in an evaluation.
Key facts
As stated in the sources, with where to find them.
- The dataset asks whether untrusted tool output can induce a random tool call unrelated to the legitimate user request. Attack goals are deliberately non-malicious to isolate instruction-hierarchy behavior.Sections 1–3
- The Agentic Optimizer uses collaborating agents to search adaptively; results vary with attacker capability and the defender-query budget.Agentic Optimizer; experimental results
- The Gemini experiments disable safety classifiers for both attacker and defender models, as well as outside-model prompt-injection mitigations. Results do not evaluate all safeguards of a deployed Gemini product.Section 5.1: evaluation setup
- The simplified next-tool-call setting does not measure a full intrusion, exfiltration or multi-step production compromise.Dataset design and limitations
Findings that cite this record
Key questions this bears on
- Can prompt injection against AI agents be reliably defended?Not reliably. Bounding what untrusted input can trigger helps, but trust boundaries must survive delegation and tool execution.
- How far can measured AI cyber capability be trusted?As a lower or conditional bound. Scores move with budget, pipeline and contamination, and benchmark names or labels can hide different setups.
Sources
Related records
Oct 27, 2026
Oct 22, 2026
Oct 20, 2026
Oct 16, 2026
Oct 13, 2026
Oct 12, 2026