Chronicle/Capability & gating

CAISI evaluation finds DeepSeek models lag US models on cyber tasks and are far easier to hijack

Sep 30, 2025CapabilityEvaluation reportSignificance assistant-drafted

NIST's CAISI evaluated DeepSeek R1, R1-0528 and V3.1 against US reference models across 19 benchmarks, as directed by the AI Action Plan. CAISI reports the largest capability gap on software engineering and cyber tasks, and found DeepSeek-based agents far more likely to follow hijacking instructions and to comply with jailbroken malicious requests.

Why it matters

It is a government evaluation that treats agent hijacking susceptibility as a national security property of foreign models.

Key facts

As stated in the sources, with where to find them.

On software engineering and cyber tasks, the best US model evaluated solves over 20% more tasks than the best DeepSeek model.NIST news release, key findings
Agents built on DeepSeek R1-0528 were on average 12 times more likely than evaluated US frontier models to follow malicious hijacking instructions in simulated environments.NIST news release, security findings
With a common jailbreak, R1-0528 responded to 94% of overtly malicious requests versus 8% for US reference models.NIST news release, security findings
US reference models: GPT-5, GPT-5-mini, gpt-oss (OpenAI) and Opus 4 (Anthropic).NIST news release

Findings that cite this record

No tracked finding cites this record yet.

Key questions this bears on

How far can measured AI cyber capability be trusted?As a lower or conditional bound. Scores move substantially with token budget, evaluation pipeline, and benchmark contamination.

Sources

CAISI Evaluation of DeepSeek AI Models Finds Shortcomings and RisksNISTSep 30, 2025Primary sourceAccessed Sep 25, 2026

Related records

May 1, 2026

CAISI evaluation finds DeepSeek V4 Pro trails US frontier models by about eight months

CapabilityEvaluation reportUS Center for AI Standards and Innovation, DeepSeek, NIST

Mar 16, 2026

CAISI, UK AISI and Gray Swan competition finds concealed indirect injections succeed on all 13 frontier models

DefensePaperGray Swan AI, US Center for AI Standards and Innovation, UK AI Security Institute

Nov 24, 2025

Anthropic reports 1.4% prompt injection success for Claude Opus 4.5 with improved Chrome extension safeguards

DefenseEvaluation reportAnthropic, Gray Swan AI

Mar 13, 2026

Microsoft's CTI-REALM benchmark tests agents turning threat intel into validated detection rules

DefenseBenchmarkMicrosoft

Mar 10, 2026

OpenAI releases IH-Challenge RL dataset and reports instruction hierarchy gains on injection benchmarks

DefenseDatasetOpenAI

Mar 1, 2026

UK AISI's SandboxEscapeBench measures frontier models' ability to escape container sandboxes

DefenseBenchmarkUK AI Security Institute, University of Oxford

Organizations: US Center for AI Standards and Innovation, NIST, DeepSeek
Topics: Capability evaluation Prompt injection Open-weight diffusion
Attack surface: Agent model Untrusted content
Methods: CTF and task benchmarks Jailbreaking safeguards
Benchmarks and tools: Claude Opus 4.x DeepSeek models GPT-5 family
Fide research: FID-075: Cybersecurity capabilities and whole-system risk
Record: caisi-deepseek-evaluation-2025 · added 2026-09-25

Cite this record

Fide AI. "CAISI evaluation finds DeepSeek models lag US models on cyber tasks and are far easier to hijack." Agentic Cyber Explorer, record caisi-deepseek-evaluation-2025. https://agentic-cyber-explorer.pages.dev/events/caisi-deepseek-evaluation-2025/ (corpus as of 2026-09-26).

BibTeX

@misc{fide-ace-caisi-deepseek-evaluation-2025,
  author = {{Fide AI}},
  title = {CAISI evaluation finds DeepSeek models lag US models on cyber tasks and are far easier to hijack},
  howpublished = {Agentic Cyber Explorer},
  year = {2026},
  url = {https://agentic-cyber-explorer.pages.dev/events/caisi-deepseek-evaluation-2025/},
  note = {Record caisi-deepseek-evaluation-2025; corpus as of 2026-09-26}
}

Data: JSON · CC BY 4.0

Spotted an error? Suggest a correction.