Researchers at Georgian and the University of Toronto ran Vercel Labs' open-source DeepSec harness with 24 model–runner configurations on a production TypeScript codebase whose vulnerabilities they say were non-public at scan time. No configuration covered all 265 confirmed security root causes, and moving from a minimal prompt to an engineered prompt with CodeQL-derived call-graph context raised criticality-weighted yield by 20–86% for three smaller or cheaper models but by 2% for GPT-5.5. The paper is published in the 2026 CAMLIS proceedings (PMLR 349).
It measures, on a codebase whose vulnerabilities were non-public at scan time, that the value of harness context depends on the model and that single runs of single models under-count findings, which bears on how vulnerability-discovery results are compared.
Key facts
As stated in the sources, with where to find them.
- 24 model–runner configurations, two runs each, scanned about 150 TypeScript candidate files of a production LLM API-gateway codebase (codename Conduit) at one pinned commit. The repository identity is withheld for disclosure; its source is public, so prior exposure to the code cannot be excluded.Section 3; Section 10.1
- Across 265 confirmed security root causes (91 rated critical or high), the strongest two-run configuration, GLM-5.3, found 160 (60.4%), and 38 (14.3%) were found by exactly one model.Section 4.1; Section 5
- From the minimal prompt (A0) to the engineered prompt with CodeQL-derived context, two-run mean security yield rose for Gemma-4-31B from 13.0 to 25.5, GLM-5.1 from 30.5 to 42.0 and Composer-2.5 from 72.5 to 87.0, and stayed at 80.0 for GPT-5.5. Criticality-weighted lifts were +86%, +36%, +20% and +2%.Section 4.2; Table 3; Figure 1
- The headline comparison changes prompt and context together. Against the engineered prompt alone (A1), the CodeQL context block raised yield only for Gemma-4-31B and lowered it for GPT-5.5, Composer-2.5 and GLM-5.1.Section 7; Section 17
- False-report rate moved separately from yield from A0 to the CodeQL context arm: GPT-5.5 10.0% to 5.3%, Composer-2.5 18.4% to 18.3%, Gemma-4-31B 32.3% to 32.9%, GLM-5.1 13.3% to 29.4%.Table 3
- On 21 public CyberGym Level-0 C/C++ instances, target recall rose with the context for all four models (Composer-2.5 17 to 19, GPT-5.5 14 to 15, GLM-5.1 8 to 10, Gemma-4-31B 4 to 5). The authors call this corroboration on a public, patched corpus, not primary evidence.Section 4.3; Table 5
- Per-model run-to-run Jaccard overlap was 0.35–0.83, so one run recovered about 68–91% of a model's two-run set. For GPT-5.5 the primary context arm's change against A0 had opposite signs in the two runs (88 to 78 and 72 to 82; Table 10).Section 5; Section 16; Table 10
- Model self-revalidation discarded valid findings: in the 12 configurations audited, of 1,232 explicit model exclusions the authors' review judged 680 to be true positives, though many were duplicates of findings kept elsewhere in the same run.Section 4.1; Section 13; Table 8
- Author-run limits: severities are model-assigned and, per maintainer feedback, typically overstated; counts describe what the pipeline confirmed, not maintainer-adjudicated severity. Cross-model results also reflect different runners and reasoning settings, and costs mix billed and API-equivalent figures that exclude review and proof work.Section 4.1; Section 7; Section 8; Section 14
Findings that cite this record
Key questions this bears on
- How far can measured AI cyber capability be trusted?As a lower or conditional bound. Scores move with budget, pipeline and contamination, and the same benchmark name can hide different setups.
- Is AI shifting the balance between finding and fixing vulnerabilities?Discovery is ahead. Verification, triage, coordinated disclosure and safe repair remain bottlenecks between AI findings and deployed fixes.