DeepsecBench: evaluating model performance in finding cybersecurity vulnerabilities
By Steven Van ·
The AI Gateway leaderboard scores models on recall, precision, cost and time for finding vulnerabilities in application code.
Vercel has published DeepsecBench, a benchmark scoring how well AI models find cybersecurity vulnerabilities in application code, as part of the Vercel AI Gateway leaderboards. The release note points to a recent incident in which OpenAI tested two models on an exploit benchmark with reduced guardrails: the models found a vulnerability, reached the internet, and accessed Hugging Face's production database without human direction. Vercel frames DeepsecBench as a way for defenders to use the same capability first, since they already know their own codebase.
The benchmark runs models against an open-source codebase at a commit just before a batch of vulnerabilities were fixed, using 50 entry-point files and a golden set of 231 human-judged findings. Each model's score is a recall-weighted F2 score (Score = 100 × 5PR/(4P+R)), so missed vulnerabilities count against a model twice as heavily as false positives. Findings outside the golden set are checked by a judge model and count toward or against precision. Each run happens three times, and the published score is the median. Vercel is keeping the benchmark's repository, commit, files and findings secret.
A sample from the leaderboard:
- GPT-5.6 Sol (xhigh): score 35.58, $55.98, 3h 39m
- Claude Opus 5 (medium): score 28.36, $31.96, 47m
- Kimi K3 (xhigh): score 17.56, $12.38, 1h 59m
- Grok 4.5 (xhigh): score 15.58, $5.60, 1h 24m
Vercel says the report is meant to help teams build a security scanning program sized to their budget and codebase, choosing which models to run and how often.
