Researchers from Fullive-AI, Peking University, JD.com and other institutions replay AgentDojo episodes up to the step where an agent reads an injected tool output, then resample the rest many times to separate what is fixed at read time from what is decided afterwards. They report that a Qwen3-8B agent without thinking is largely decided at read time, while with thinking part of the decision forms during reasoning. Zero-shot chain-of-thought monitors on cheaper models miss much of what is already decided, while stronger monitors and a lookahead monitor that judges simulated next actions come close to the measured ceiling.
It indicates that read-time injection monitors and single-run benchmark labels can misjudge reasoning agents, and that turning on thinking lowered injection success for one model but raised it for two others.
Key facts
As stated in the sources, with where to find them.
- The analysis covers 18,091 sampled branches and 4,872 prefix continuations, replaying AgentDojo (v1) episodes with byte-identical prompts up to the step where the agent first reads the injected tool output.Section 1; Section 3.1
- For Qwen3-8B across four AgentDojo suites, the share of outcome variance fixed when the injection is read fell from 0.91 without thinking to 0.62 with thinking (paired drop 95% CI −0.38 to −0.22; official label 0.89 to 0.57), and mixed states rose from 7% to 28%.Section 5.2; Table 3
- Thinking lowered Qwen3-8B's attack success rate (0.42 to 0.32, four suites) but raised it for Llama-3.1-Nemotron-Nano-8B (0.06 to 0.51, banking) and Gemini 2.5 Flash (0.62 to 0.80, banking and slack).Section 5.2; Table 3
- At 30%, 50% and 70% of the reasoning, the prefix determined 0.36, 0.55 and 0.62 of the variance left open at read time (Qwen3-8B, banking).Section 5.3
- Zero-shot chain-of-thought monitors on GPT-4o and DeepSeek-v4-flash without thinking fell significantly short of the decision ceiling in 15 of 15 cells; GPT-5, Claude Opus 4.8 and DeepSeek-v4-flash with thinking in 3 of 18. A lookahead monitor with a GPT-5 judge reading simulated next calls was never significantly below it, but needs sampling access to the agent and, for multi-turn checks, a sandbox.Section 5.4; Table 4
- In the standard tool-calling layout there was no early read-time window: 29% of complying banking episodes acted in the same step that read the injection. Moving the tool block after the newest observation created a window and lowered attack success from 48% to 29% (Qwen3-8B without thinking, banking; official label 34% to 19%); the authors say results from that layout should not be carried over.Section 5.1; Table 2; Section 6
- Two runs of the same task agreed in 0.90–0.99 of cases without thinking and 0.65–0.94 with it for the Qwen3, Nemotron and Llama-3.1 agents (Gemini 0.77 and 0.81); a majority label right with 90% probability took 3–11 runs per task for Qwen3-8B with thinking on three suites.Section 5.6; Table 6
- Author-run limits: one benchmark with public attack templates; agents are mostly 3B to 14B open models plus Gemini 2.5 Flash; exposure states come from Qwen3-8B episodes; two of four claims fixed in advance failed, and the strong and lookahead monitors were added afterwards as exploratory; defenses that act inside the reasoning were not tested; fraction-based AUCs flatter monitors, and the comply label is the authors' own.Table 1; Limitations; Section 6
Findings that cite this record
Key questions this bears on
- Can prompt injection against AI agents be reliably defended?Not reliably. Bounding what untrusted input can trigger helps, but trust boundaries must survive delegation and tool execution.
- Can AI agents defend and oversee systems on their own?Not reliably. Some designs improve benchmark triage, but autonomous defense and oversight lack broad operational validation.