# CWE-bench, by Collinear AI

> CWE-bench is a defensive cybersecurity benchmark from Collinear AI that measures whether frontier coding agents can find and fix real vulnerabilities in real codebases. On the v1 held-out set of 120 audit-and-patch tasks across 73 CWEs, graded by a deterministic programmatic verifier with a three-judge agentic panel reported alongside, the best model solves 71%. Collinear builds the training data and RL environments that move that number.

## Who this is for

Post-training, evaluation, and data-procurement teams at frontier AI labs building or hardening coding agents, and anyone deciding whether an agent can be trusted with real security work.

## Why the results are credible

- Every ranked score is a deterministic programmatic verdict: a proof-of-concept exploit must no longer work and every pre-existing test must still pass. There is no partial credit and no LLM in the ranking gate.
- v1 adds a panel of three independent LLM judges from different model families. Each judge gets a shell in a fresh copy of the patched repository, checks the fix against a fixed checklist (is the weakness closed, does legitimate behavior still work, does the fix hold against variants, was the change made honestly, does the project still run), and passes it only if every item holds; the panel passes a rollout on a majority. Its pass@1 is reported beside the programmatic score, not in place of it.
- The 120-task evaluation set is held out and private to Collinear, so it cannot be trained against or memorized. The scores reflect real generalization, not contamination.
- The v0 results (100 tasks, 54 CWEs, deterministic verifier only, independently evaluated by Artificial Analysis) remain available on the site under the v0 switch.

## How it works

An agent gets a checkout of a real open-source repository and one instruction: audit the code and fix what you find. Nobody hands it the CVE, so it has to locate the vulnerability before it can patch it, the way a real security audit works.

## Key results (v1, deterministic pass@1; judge-panel pass@1 in brackets)

- Leaderboard, twelve models (rounded to the nearest point; models a point or two apart are effectively tied): Gemini 4 Carbon 71% [71%], no cost figure yet; Grok 4.7 68% [64%] at $2.75 per rollout; Gemini 4 Argon 68% [62%] at $6.63 (priced from its token counts at $2 per million input, $0.20 per million cached input and $10 per million output tokens, thinking included); GPT-6 Astra 68% [66%] at $2.85; Claude Opus 5.5 67% [67%] at $0.79; Claude Fable 5.1 58% [60%] at $3.47; Grok 4.6 57% [55%] at $1.09; DeepSeek-V4.1-Flash 55% [53%] at $0.09; Muse Spark 1.3 55% [51%] at $3.95; Hy4 Preview 53% [50%] at $0.47; GPT-6 Sol 52% [52%] at $0.96; Inkling 37% [33%] at $3.00.
- Pareto frontier, cheapest first: DeepSeek-V4.1-Flash, Claude Opus 5.5, Grok 4.7, GPT-6 Astra, Gemini 4 Argon (a member must gain at least a point on the next-cheapest, and models within a point of the top score are tied for the lead and all count; Hy4 Preview scores lower than DeepSeek at five times the cost). 74x the cost buys 1.2x the pass rate. Gemini 4 Carbon has no cost figure yet, so it is not on the frontier.
- pass@4 (any of a task's four rollouts): Gemini 4 Carbon 81%, Grok 4.7 81%, Claude Opus 5.5 79%, Gemini 4 Argon 75%, Hy4 Preview 75%, GPT-6 Astra 74%, DeepSeek-V4.1-Flash 74%, Claude Fable 5.1 73%, Grok 4.6 72%, Muse Spark 1.3 69%, GPT-6 Sol 68%; Inkling not reported.
- Four models ran on both v0 and v1. Deterministic pass@1 on the same 100 tasks (v1's 20 new tasks left out): GPT-6 Astra 52% -> 68%, Claude Fable 5.1 53% -> 62%, Hy4 Preview 34% -> 54%, Inkling 16% -> 38%. The lift reflects verifiers made fairer with the judge panel's help (probes reviewed wherever they failed a correct patch), not models cracking the benchmark.
- 5 of the 120 tasks are unsolved by every one of the 12 models evaluated (judge panel); 40 are solved by every model.
- Weakest OWASP 2025 categories across the board: A08 Software or Data Integrity Failures (37% mean), A06 Insecure Design (50%), A07 Authentication Failures (52%). Strongest: A04 Cryptographic Failures (76%).
- By language, C/C++ (62% mean) and JS/TS (62%) lead; Java (54%) and Rust (54%, three tasks) trail.
- 73 CWEs mapped to the OWASP Top 10 (ten to fourteen tasks per category), across C/C++, Go, Java, JavaScript/TypeScript, Python, and Rust.

## What Collinear offers

Collinear AI builds hard, verifiable RL environments and training data for frontier labs, spanning coding, computer use, and enterprise tool use. For defensive security specifically:

- A separate 1,000+ task training corpus of real audit-and-patch tasks, built to raise an agent's ability to find and fully remediate vulnerabilities across larger repositories and multiple CWEs.
- Model evaluation on the held-out CWE-bench set.
- Environments calibrated to the model you are training, shipping the tasks it currently fails.

If you are training a coding agent and want it to pass more of these, the training corpus is the product.

## Talk to us

- Request the training corpus or a model evaluation: cyber@collinear.ai
- Paper: https://arxiv.org/abs/2610.17020
- Site: https://cwe-bench.com/
- Leaderboard: https://cwe-bench.com/#leaderboard
- v0 results: https://cwe-bench.com/?v=v0