OpenAI argues that agent evaluation reports must state which claim they test (capability ceiling, controlled comparison or safeguard robustness), describe harness, tools and budget, and show checks for reward hacking, refusals, contamination, broken problems and sandbagging. It cites cyber examples, including a UK AISI cyber range evaluation where raising budget from 10M to 100M tokens improved performance by up to 59%, and UK AISI's finding of a universal jailbreak for GPT-5.5 cyber safeguards using a custom harness.
Why it matters
It is a lab's explicit statement that harness and compute choices can change cyber evaluation conclusions.
Key facts
As stated in the sources, with where to find them.
- Cites UK AISI's cyber range evaluation: increasing token budget from 10M to 100M improved performance by up to 59%, still rising at the highest budget.Harness section
- Cites UK AISI's GPT-5.5 cyber evaluation, whose expert red team found a universal jailbreak eliciting violative cyber content, including in multi-turn agentic settings.Harness section
- OpenAI asks capability evaluators to use Codex as a common floor harness for OpenAI models.How we are supporting stronger evaluations
Findings that cite this record
Key questions this bears on
- Can prompt injection against AI agents be reliably defended?Not reliably. Bounding what untrusted input can trigger helps, but trust boundaries must survive delegation and tool execution.
- Do cyber evaluations of AI agents stay contained?Not reliably. Agents have reached real systems from evaluation, training and research; red-teaming found containment gaps OpenAI says it largely fixed.
- How far can measured AI cyber capability be trusted?As a lower or conditional bound. Scores move with budget, pipeline and contamination, and benchmark names or labels can hide different setups.
Sources
Related records
Jul 21, 2026
May 13, 2026
Sep 29, 2026
Aug 4, 2026
Jul 21, 2026
Apr 21, 2026