Collinear released CWE-bench v2 with 100 public repository security-repair tasks covering 91 distinct CWEs, alongside held-out evaluation. Tasks check security and existing functionality, with programmatic graders and human review. Public-set results and the separate combined leaderboard use different denominators, and neither is a direct continuation of v1 scores.
Why it matters
It improves inspection of defensive agent evaluations while keeping benchmark version, task selection and execution harness visible.
Key facts
As stated in the sources, with where to find them.
- The v2 public set has 100 tasks covering 91 CWEs. Access requires accepting the dataset’s conditions; the grader and harness are available for inspection.Release: introduction; Run it from the Hub
- Collinear describes LLM-assisted verifier curation and manual review of every task, checking security behavior and preservation of original functionality. These are benchmark checks, not guarantees for deployed patches.Release: quality checklist
- The separate CWE-bench site describes 150 held-out tasks in addition to the 100 public tasks, for 250 tasks and 185 distinct CWEs overall. Public-set and combined leaderboard figures must retain their task-set labels.CWE-bench site: benchmark overview
- Scores depend on version, agent harness and attempt budget. The public v2 set and combined leaderboard should not replace the earlier private v1 record’s results.Release: Results; CWE-bench site: methodology
Findings that cite this record
Key questions this bears on
- How far can measured AI cyber capability be trusted?As a lower or conditional bound. Scores move with budget, pipeline and contamination, and benchmark names or labels can hide different setups.
- Is AI shifting the balance between finding and fixing vulnerabilities?Discovery is ahead. Verification, triage, coordinated disclosure, safe repair and getting fixes to users remain bottlenecks between AI findings and protection.
Sources
Related records
Oct 12, 2026
Sep 28, 2026
Oct 28, 2026
Oct 25, 2026
Oct 13, 2026
Oct 5, 2026