As a lower or conditional bound. Scores move with budget, pipeline and contamination, and benchmark names or labels can hide different setups.
UK AISI reports that fixed, low token budgets understate frontier cyber capability and its rate of progress. A preprint audit finds pipeline choices alone can move cyber benchmark scores by tens of points, public CTF benchmarks can be contaminated, and counting a crash as exploitation overstates capability. Anthropic's September 2026 Claude Opus 5.5 system card states that it rewrote many cyber harnesses, so its numbers differ from earlier cards, and its report on GLM-5.3 gives ExploitBench counts out of 410 attempts without the trials, arms or harness that make the same denominator mean something different in the Opus 5.5 card (301 of 410 there is 41 bugs times five trials times two arms). Its ExploitGym figures, 289 exploits within two hours and 300 within six for Opus 5.5 against 203 and 264 for Mythos 5.1, show that a longer wall-clock budget helped one model far more than the other, and a released model reaching full code execution in 73.4% of ExploitBench runs narrows the May 2026 finding that most public models stalled before code execution. A study of 24 model setups on a production codebase whose vulnerabilities were non-public at scan time adds that prompt and context changes raised criticality-weighted vulnerability-discovery yield by 20–86% for three smaller models but by 2% for GPT-5.5, and that one run recovered only 68–91% of a model’s two-run findings.
Task labels and cost accounting matter as well. Google’s 2026-10-28 Gemini 4 Carbon announcement presents an 88.2% SEC-bench Pro score as a discovery result, but the benchmark’s authors describe known-vulnerability tasks that supply source paths, vulnerability class and expected error, so the score is not a blind discovery rate, and the announcement does not establish matching harness or grading settings. Epoch AI reports about $1,000 in marginal API cost per Firefox release for GLM-5.3’s exploit chains after more than $100,000 in total Firefox research spend, and no Chrome success at about $100,000, so a per-target cost depends on which earlier spending is counted. Google Project Zero reports that BigSleep rediscovered about 62% of 112 selected historical V8 bugs over 25 attempts per introducing commit, a controlled rediscovery figure that measures something different from its production comparison with a fuzzer. Single capability numbers, including trend estimates, are best read as lower or conditional bounds, and a number is only comparable when its budget, harness, denominator and the task information supplied to the model are stated.
The findings behind it
4 reported, 1 qualifiedEach finding carries a status that changes as new work arrives. What the statuses mean.
- Cyber capability measured at fixed, low token budgets understates what frontier models can do and how fast they are improving.Reported measured · 6 evidence records
- Evaluation pipeline choices alone can move a model's cybersecurity benchmark score by more than 80 points and reorder models.Reported measured · 7 evidence records
- Scores on public CTF benchmarks can be inflated when agents find published solutions, and static benchmarks lose validity as their flaws are patched.Reported measured · 2 evidence records
- Counting a crash as exploitation success overstates capability; most public models in May 2026 stalled before code execution on a browser engine.Reported measured · 3 evidence records
- On ExploitGym (May 2026), the strongest agents produced working exploits for 157 and 120 of 898 instances with mitigations off; with standard mitigations on, 45 and 21 survived.Qualified measured · 3 evidence records
Answer history
Answers are never edited after the fact. A revision adds a new answer and keeps the earlier ones here.
- 2026-11-01moderate confidencecurrentAs a lower or conditional bound. Scores move with budget, pipeline and contamination, and benchmark names or labels can hide different setups.Google’s Gemini 4 Carbon announcement, Epoch AI’s Firefox patch-gap study and Project Zero’s BigSleep comparison show task labels, supplied information, cost accounting and attempt counts shaping reported numbers, extending the comparability rule; the conclusion and confidence are unchanged.
- 2026-10-22moderate confidenceAs a lower or conditional bound. Scores move with budget, pipeline and contamination, and the same benchmark name can hide different setups.A vulnerability-discovery study shows harness context and run-to-run variation changing measured yield, with effects that differ by model.
- 2026-09-30moderate confidenceAs a lower or conditional bound. Scores move with budget, pipeline and contamination, and the same benchmark name can hide different setups.Added two Anthropic reports from September 2026 (records anthropic-opus-5-5-system-card-cyber-2026 and anthropic-glm-5-3-cyber-assessment-2026) that show the same benchmark name and denominator hiding different setups, and a case where the effect of budget varies by suite. The conclusion is unchanged and now carries a concrete comparability rule.
- 2026-09-25moderate confidenceAs a lower or conditional bound. Scores move substantially with token budget, evaluation pipeline, and benchmark contamination.First answer, drawn from the findings linked here.