Findings/fixed-budgets-understate-cyber-capability

Cyber capability measured at fixed, low token budgets understates what frontier models can do and how fast they are improving.

Reportedmeasured4 evidence records from 2 independent sourcesassistant-drafted

Scope: what this does not show

UK AISI measurements on its task suite; the size of the effect varies by model and task.

Reported: Stated by one source and not yet corroborated or challenged.

Evidence

May 29, 2026

OpenAI publishes a playbook on harness choice and validity checks for third-party evaluations

OpenAI

Jul 2, 2026

UK AISI finds agent evaluations understate cyber capability without accounting for test-time compute

UK AI Security Institute

Raising the budget from 2.5M to 50M tokens moved one model's cyber time horizon from about 40 minutes to about 4 hours.

Mar 1, 2026

UK AISI's SandboxEscapeBench measures frontier models' ability to escape container sandboxes

arXiv

May 13, 2026

UK AISI says frontier cyber task horizons doubled every 4.7 months, with Mythos Preview and GPT-5.5 above trend

UK AI Security Institute

UK AISI's May post already said the 2.5M-token cap understates frontier capability.

How it relates to other findings

supportsqualifiescontestssupersedes

ReportedCorroboratedQualifiedContestedSupersededRevalidate· node size = evidence records · columns group by topic

Select a finding to see how it relates to others. Arrows point from the newer finding to the one it supports, qualifies, contests, or supersedes.

This finding qualifies Under a 2.5M-token cap, frontier cyber task time horizons doubled on the order of months between late 2024 and early 2026.The doubling estimate was measured at a fixed low budget.

Key questions that rely on this finding

How far can measured AI cyber capability be trusted?As a lower or conditional bound. Scores move substantially with token budget, evaluation pipeline, and benchmark contamination.

Status history

2026-05-29ReportedOpenAI's evaluation playbook warns that unreported budgets understate capability. · record
2026-07-02CorroboratedUK AISI measures the effect directly. · record
2026-09-25ReportedcorrectionThe OpenAI playbook cites UK AISI's own measurements, so all evidence comes from one evaluator. · record

What would change this: Evidence that capability plateaus at budgets in common use.
Evidence kind: measured: An experiment or evaluation produced the number.
Freshness: Evidence 86d old · recheck in 184 daysNewest evidence Jul 2, 2026. Half-life 270 days.
Methods tested: Compute-scaled evaluation CTF and task benchmarks
Topics: Evaluation validity Capability evaluation
Fide research: FID-075: Cybersecurity capabilities and whole-system risk

Cite this record

Fide AI. "Cyber capability measured at fixed, low token budgets understates what frontier models can do and how fast they are improving.." Agentic Cyber Explorer, record fixed-budgets-understate-cyber-capability. https://agentic-cyber-explorer.pages.dev/findings/fixed-budgets-understate-cyber-capability/ (corpus as of 2026-09-26).

BibTeX

@misc{fide-ace-fixed-budgets-understate-cyber-capability,
  author = {{Fide AI}},
  title = {Cyber capability measured at fixed, low token budgets understates what frontier models can do and how fast they are improving.},
  howpublished = {Agentic Cyber Explorer},
  year = {2026},
  url = {https://agentic-cyber-explorer.pages.dev/findings/fixed-budgets-understate-cyber-capability/},
  note = {Record fixed-budgets-understate-cyber-capability; corpus as of 2026-09-26}
}

Data: JSON · CC BY 4.0

Know of evidence that changes this? Tell us.