One preprint audit of public cyber benchmarks. Anthropic's September 2026 Opus 5.5 card states that rewritten harnesses changed its numbers relative to earlier cards without quantifying the change. A 2026 vulnerability-discovery study adds that harness context changes discovery yield on one production codebase differently by model; it measures yield, not a benchmark score. Google’s Gemini 4 Carbon announcement (2026-10-28) shows a related task-label mismatch, presenting a known-vulnerability score as discovery, not a measured score change.
Reported: Stated by one source and not yet corroborated or challenged.
Evidence
The same ExploitBench name and a matching 410-run denominator appear in two Anthropic reports with different or unstated setups, so the counts cannot be compared without the harness details.
Anthropic states that it rewrote many cyber harnesses and reran older models, so Opus 5.5 numbers differ from earlier system cards; the size of the change is not quantified.
On a production codebase whose vulnerabilities were non-public at scan time, prompt and context changes raised criticality-weighted discovery yield by 20–86% for three smaller models and 2% for GPT-5.5, and one run recovered 68–91% of a model’s two-run findings.
Google presents an 88.2% SEC-bench Pro score as a discovery result, though the benchmark’s known-vulnerability tasks supply hints, and the announcement does not establish matching harness or grading settings.
Related comparison limit: Anthropic says each lab’s CyScenarioBench score uses unique tasks; the release table does not establish a matched-task cross-lab ranking.
Related comparison limit: v2 public-set scores and the combined public/held-out leaderboard use different task populations; v1 remains a separate benchmark version.
How it relates to other findings
Select a finding to see how it relates to others. Arrows point from the newer finding to the one it supports, qualifies, contests, or supersedes.
- Independent government testing can find a model weaker on agentic tasks than its developer's self-reported benchmarks suggest. supports this findingBoth show headline scores depend on evaluation choices; here the difference is which benchmarks are run, not how the same benchmark is run.
Key questions that rely on this finding
- How far can measured AI cyber capability be trusted?As a lower or conditional bound. Scores move with budget, pipeline and contamination, and benchmark names or labels can hide different setups.
Status history
- 2026-09-08ReportedAudit of pipeline dependence in cyber benchmarks. · record