# CWE-bench, by Collinear AI

> CWE-bench is a defensive cybersecurity benchmark from Collinear AI that measures whether frontier coding agents can find and fix real vulnerabilities in real codebases. On a held-out set of 100 audit-and-patch tasks across 54 CWEs, the best model solves 47.8%, independently evaluated and published by Artificial Analysis. Collinear builds the training data and RL environments that move that number.

## Who this is for

Post-training, evaluation, and data-procurement teams at frontier AI labs building or hardening coding agents, and anyone deciding whether an agent can be trusted with real security work.

## Why the results are credible

- Independently evaluated and published by Artificial Analysis, not self-reported by the vendor.
- Graded by a deterministic programmatic verifier: a task is solved only when the exploit is blocked and every pre-existing test still passes. No partial credit, and no LLM judge in the leaderboard gate, so the reward cannot be farmed.
- The 100-task evaluation set is held out and private to Collinear and Artificial Analysis, so it cannot be trained against or memorized. The scores reflect real generalization, not contamination.

## How it works

An agent gets a checkout of a real open-source repository and one instruction: audit the code and fix what you find. Nobody hands it the CVE, so it has to locate the vulnerability before it can patch it, the way a real security audit works.

## Key results (pass@1, deterministic verifier)

- Best model: 47.8% (Claude Fable 5), at an average API cost of $10.27 per rollout. The cheapest model above 30% is DeepSeek-V4-Flash at $0.13.
- 10 of the 100 tasks are unsolved by every one of the 14 models evaluated.
- 54 CWEs mapped to the OWASP Top 10, across C/C++, Go, Java, JavaScript/TypeScript, Python, and Rust.

## What Collinear offers

Collinear AI builds hard, verifiable RL environments and training data for frontier labs, spanning coding, computer use, and enterprise tool use. For defensive security specifically:

- A separate 1,000+ task training corpus of real audit-and-patch tasks, built to raise an agent's ability to find and fully remediate vulnerabilities across larger repositories and multiple CWEs.
- Model evaluation on the held-out CWE-bench set.
- Environments calibrated to the model you are training, shipping the tasks it currently fails.

If you are training a coding agent and want it to pass more of these, the training corpus is the product.

## Talk to us

- Request the training corpus or a model evaluation: info@collinear.ai
- Site: https://cwe-bench.com/
- Leaderboard: https://cwe-bench.com/#leaderboard
- Independent results, Artificial Analysis: https://artificialanalysis.ai/
- Collinear AI: https://collinear.ai/

## How to cite

@misc{cwebench2026, title = {CWE-bench: a defensive cybersecurity benchmark for coding agents}, author = {Collinear AI}, year = {2026}, howpublished = {\url{https://cwe-bench.com}}}