Findings/independent-tests-differ-from-self-reports

Independent government testing can find a model weaker on agentic tasks than its developer's self-reported benchmarks suggest.

Reportedmeasured1 evidence record from 1 independent sourceassistant-drafted

Scope: what this does not show

One evaluation of one model on non-public benchmarks.

Reported: Stated by one source and not yet corroborated or challenged.

Evidence

May 1, 2026

CAISI evaluation finds DeepSeek V4 Pro trails US frontier models by about eight months

NIST

CAISI found weaker results on non-public agentic and reasoning benchmarks (ARC-AGI-2 semi-private, PortBench, CTF-Archive-Diamond) than DeepSeek self-reported.

How it relates to other findings

supportsqualifiescontestssupersedes

ReportedCorroboratedQualifiedContestedSupersededRevalidate· node size = evidence records · columns group by topic

Select a finding to see how it relates to others. Arrows point from the newer finding to the one it supports, qualifies, contests, or supersedes.

This finding supports Evaluation pipeline choices alone can move a model's cybersecurity benchmark score by more than 80 points and reorder models.Both show headline scores depend on evaluation choices; here the difference is which benchmarks are run, not how the same benchmark is run.

Status history

2026-05-01ReportedCAISI evaluation of DeepSeek V4 Pro. · record

What would change this: Agreement between independent and self-reported results across several models.
Evidence kind: measured: An experiment or evaluation produced the number.
Freshness: Evidence 148d old · recheck in 122 daysNewest evidence May 1, 2026. Half-life 270 days.
Methods tested: CTF and task benchmarks
Topics: Evaluation validity Capability evaluation
Fide research: FID-075: Cybersecurity capabilities and whole-system risk

Cite this record

Fide AI. "Independent government testing can find a model weaker on agentic tasks than its developer's self-reported benchmarks suggest.." Agentic Cyber Explorer, record independent-tests-differ-from-self-reports. https://agentic-cyber-explorer.pages.dev/findings/independent-tests-differ-from-self-reports/ (corpus as of 2026-09-26).

BibTeX

@misc{fide-ace-independent-tests-differ-from-self-reports,
  author = {{Fide AI}},
  title = {Independent government testing can find a model weaker on agentic tasks than its developer's self-reported benchmarks suggest.},
  howpublished = {Agentic Cyber Explorer},
  year = {2026},
  url = {https://agentic-cyber-explorer.pages.dev/findings/independent-tests-differ-from-self-reports/},
  note = {Record independent-tests-differ-from-self-reports; corpus as of 2026-09-26}
}

Data: JSON · CC BY 4.0

Know of evidence that changes this? Tell us.