Chronicle/Defense & research

CWE-bench v2 releases inspectable security-repair tasks alongside held-out tests

DefenseBenchmarkSignificance assistant-drafted

Collinear released CWE-bench v2 with 100 public repository security-repair tasks covering 91 distinct CWEs, alongside held-out evaluation. Tasks check security and existing functionality, with programmatic graders and human review. Public-set results and the separate combined leaderboard use different denominators, and neither is a direct continuation of v1 scores.

Why it matters

It improves inspection of defensive agent evaluations while keeping benchmark version, task selection and execution harness visible.

Key facts

As stated in the sources, with where to find them.

  • The v2 public set has 100 tasks covering 91 CWEs. Access requires accepting the dataset’s conditions; the grader and harness are available for inspection.Release: introduction; Run it from the Hub
  • Collinear describes LLM-assisted verifier curation and manual review of every task, checking security behavior and preservation of original functionality. These are benchmark checks, not guarantees for deployed patches.Release: quality checklist
  • The separate CWE-bench site describes 150 held-out tasks in addition to the 100 public tasks, for 250 tasks and 185 distinct CWEs overall. Public-set and combined leaderboard figures must retain their task-set labels.CWE-bench site: benchmark overview
  • Scores depend on version, agent harness and attempt budget. The public v2 set and combined leaderboard should not replace the earlier private v1 record’s results.Release: Results; CWE-bench site: methodology

Findings that cite this record

Key questions this bears on

Sources

Related records