Organizations/security vendor

Collinear

4 records4 defense
Nov 3, 2026
CWE-bench v2 releases inspectable security-repair tasks alongside held-out tests
DefenseBenchmarkCollinear

Collinear released CWE-bench v2 with 100 public repository security-repair tasks covering 91 distinct CWEs, alongside held-out evaluation. Tasks check security and existing functionality, with programmatic graders and human review. Public-set results and the separate combined leaderboard use different denominators, and neither is a direct continuation of v1 scores.

Oct 12, 2026
Collinear's CWE-bench: 120 private audit-and-repair tasks, with three of eleven agents tied at 68% pass@1
DefenseBenchmarkCollinear

Collinear researchers introduce CWE-bench, a private benchmark of 120 repository audit-and-repair tasks built on real open-source repositories with injected vulnerabilities across 73 CWE types, eight languages and all ten OWASP Top 10:2025 categories; agents receive a threat model but not the weakness, its location or the number of vulnerable sites. A patch passes only if a programmatic verifier finds the exploits blocked and existing functionality tests passing, and the authors check the verifier against a three-judge agentic panel. They report that Grok 4.7, Gemini 4 Argon and GPT-6 Astra tie at 68% pass@1, that Claude Opus 5.5 reaches 67%, and that pass@4 for the tied agents ranges from 74% to 81%; the results are author-run on a task set outsiders cannot inspect.

Sep 28, 2026
Artificial Analysis launches a Cyber Index and industry alliance scoring models on source-level vulnerability finding and fixing
DefenseBenchmarkArtificial Analysis, Collinear, IBM

Artificial Analysis announced a Cyber Index Alliance, with Collinear AI, IBM, NVIDIA and Vercel as launch partners, and a v1 Cyber Index that combines three source-code evaluations: repository audit and patch (CWE-Bench-AA, 120 private tasks), finding expert-verified vulnerabilities (DeepsecBench-AA) and discovering, reproducing and patching memory-safety bugs (CyberGym-E2E-AA, 131 tasks). Artificial Analysis says it excludes exploit building, runs all three on its open-source Stirrup harness, and reports safety refusals separately from scores. The article reports failure-mode statistics and says several frontier models refuse almost every CyberGym-E2E-AA task; per-model index scores appear only in charts.

Sep 2, 2026
Google releases Gemini 3.8 Flash Cyber for trusted defenders, emphasizing automated patching
DefenseTool releaseGoogle, Google DeepMind, Collinear

Google introduced Gemini 3.8 Flash Cyber, a cybersecurity-tuned model with more permissive cyber mitigations, available only to trusted defenders through a new Fairwind Program. Google says it prioritized vulnerability fixing over exploitation and reports 47.2% pass@1 on Collinear's CWE-Bench patching benchmark, over 70% on an internal 20-language discovery benchmark, and 2.6 times more correct Chrome patches than larger commercial models.