Agentic Cyber Explorerby Fide AI · PreviewPreview
Chronicle/Defense & research

Collinear's CWE-bench: 120 private audit-and-repair tasks, with three of eleven agents tied at 68% pass@1

DefenseBenchmarkSignificance assistant-drafted

Collinear researchers introduce CWE-bench, a private benchmark of 120 repository audit-and-repair tasks built on real open-source repositories with injected vulnerabilities across 73 CWE types, eight languages and all ten OWASP Top 10:2025 categories; agents receive a threat model but not the weakness, its location or the number of vulnerable sites. A patch passes only if a programmatic verifier finds the exploits blocked and existing functionality tests passing, and the authors check the verifier against a three-judge agentic panel. They report that Grok 4.7, Gemini 4 Argon and GPT-6 Astra tie at 68% pass@1, that Claude Opus 5.5 reaches 67%, and that pass@4 for the tied agents ranges from 74% to 81%; the results are author-run on a task set outsiders cannot inspect.

Why it matters

It reports repair results for frontier agents on tasks with no diagnosis given and a verifier that also checks preserved functionality, though the private task set limits independent checking.

Key facts

As stated in the sources, with where to find them.

  • Design: 120 private held-out tasks, each starting from an open-source repository with a real CVE into which vulnerabilities with no published fix are injected, spanning 73 CWE types, eight languages (C and C++, JavaScript and TypeScript counted separately) and 10 to 14 tasks per OWASP Top 10:2025 category. Agents receive the repository and an audit instruction describing the system and its threat model, work without internet access beyond the model API, and have one hour per attempt; a timed-out attempt is graded on the repository as it stands.v1, Abstract; Sections 3.1 and 3.2; Appendix A
  • Grading: an attempt passes only if the security verifier (exploits for the injected vulnerabilities blocked) and the functional verifier (existing functionality tests pass) both succeed. A task is admitted only if a reference patch passes both. Disagreements with the judge panel were manually reviewed during development and led to added probes for incomplete repairs and behavior-based checks.v1, Sections 3.1 to 3.3; Appendix D
  • Setup: eleven frontier models, each run in a coding-agent harness (Claude Code, Codex, Antigravity or OpenCode), so scores reflect model and harness together; high reasoning effort, four independent attempts per task; provider refusals count as failures, and where Claude Code's provider routed a flagged request to a fallback model the attempt is credited to the original model.v1, Section 4.1; Appendix A
  • Programmatic pass@1: Grok 4.7 68%, Gemini 4 Argon 68%, GPT-6 Astra 68%, Claude Opus 5.5 67%. For the three tied agents pass@4 is 81%, 75% and 74% respectively, and mean API cost per rollout is $2.75, $6.63 and $2.85; Claude Opus 5.5 costs $0.79 and DeepSeek V4.1 Flash reaches 55% pass@1 at $0.09. The abstract's pair of 68% and 81% combines the shared pass@1 with the highest pass@4, which belongs to Grok 4.7 alone.v1, Sections 4.2 to 4.4; Table 2
  • Grader agreement: the programmatic verifier and a panel of three agentic judges (GPT-5.4, Claude Sonnet 5 and Gemini 3.8 Flash, majority vote on five criteria) agree on 93.9% of 5,280 attempts (kappa 0.88). Of 320 disagreements, 200 are patches the programmatic verifier accepts and the panel rejects; the same four agents lead under both scores. Judges receive the vulnerability description, the reference patch and the verifier's result. In manual review the verifier's definition of a correct repair prevailed unless it was shown to be in error.v1, Sections 4.2 and 5; Appendices B to D; Table 4
  • Retries and coverage: GPT-6 Astra passes all four attempts on 74 tasks with mixed outcomes on 15, while Grok 4.7 passes all four on 63 with mixed outcomes on 34. The three tied agents pass at least once on 97, 90 and 89 tasks, 69 in common, and 110 of 120 together.v1, Section 4.3; Figure 3
  • By language and category: the leading agents trade places across both. The paper reports integrity failures (A08) as the hardest category, with a top score of 57%, and cryptographic failures (A04) as the easiest, with Claude Opus 5.5 at 90%. The claim that difficulty varies far more by category than by language is read from figures and is not backed by a reported statistical test. An item-response analysis estimates that 30 tasks (25 to 34 across bootstrap refits) are ones the strongest agent solves less than half the time.v1, Section 4.5; Figure 5; Appendix F
  • Scope and provenance: the authors say the 73 CWEs are a small fraction of the more than 900 catalogued, and that benchmark results should be reported per class. The paper does not mention Artificial Analysis, Google or a benchmark version label. It lists Gemini 4 Argon at 68% pass@1, the figure Google reported on 2026-09-30, but does not say whether Google's figure came from this task set.v1, Sections 5 and 4.2; Table 2

Findings that cite this record

No tracked finding cites this record yet.

Key questions this bears on

Sources

Related records