Findings/pipeline-choices-move-cyber-scores

Evaluation pipeline choices alone can move a model's cybersecurity benchmark score by more than 80 points and reorder models.

Reportedmeasured7 evidence records from 5 independent sourcesassistant-drafted
Scope: what this does not show

One preprint audit of public cyber benchmarks. Anthropic's September 2026 Opus 5.5 card states that rewritten harnesses changed its numbers relative to earlier cards without quantifying the change. A 2026 vulnerability-discovery study adds that harness context changes discovery yield on one production codebase differently by model; it measures yield, not a benchmark score. Google’s Gemini 4 Carbon announcement (2026-10-28) shows a related task-label mismatch, presenting a known-vulnerability score as discovery, not a measured score change.

Reported: Stated by one source and not yet corroborated or challenged.

Evidence

How it relates to other findings

supportsqualifiescontestssupersedes
ReportedCorroboratedQualifiedContestedSupersededRevalidate· node size = evidence records · columns group by topic

Select a finding to see how it relates to others. Arrows point from the newer finding to the one it supports, qualifies, contests, or supersedes.

Key questions that rely on this finding

Status history

  1. 2026-09-08ReportedAudit of pipeline dependence in cyber benchmarks. · record