Skip to the content
Know your tech before you buy it. Makers: get listed from $49/mo

CyberSecEval

meta-llama.github.io

  • Free plan
  • Open source
  • macOS
  • Linux
  • Web
  • API
  • Self-hosted

What CyberSecEval is

CyberSecEval 4 is a benchmark suite for assessing cybersecurity vulnerabilities and defensive capabilities in large language models. Its evaluations cover responses to cyberattack requests, mistaken refusals of borderline benign queries, insecure code suggestions, prompt injection, and malicious instructions aimed at code interpreters. Prompt injection tests include textual, multilingual, and visual attacks, with a judge model assessing whether an injected instruction succeeded. Secure-code checks cover instruction and autocomplete settings.

The suite also includes AutoPatchBench, which measures patch-generation agents on C and C++ vulnerabilities found through fuzzing, and CyberSOCEval, developed with CrowdStrike, for malware analysis and threat-intelligence reasoning. The project supports API, Linux, and self-hosted use. Its guide lists API support for OpenAI, Anyscale, and Together, with instructions for adding custom providers for self-hosted models. Setup uses a Python virtual environment, listed requirements, and a Python command-line module. AutoPatch requires Podman and substantial compute and storage; the project recommends at least 80 CPUs and 8 TB of storage and says Apple Silicon MacBooks are not currently supported. The benchmarks use the MIT license for research and commercial use.

Who it suits

CyberSecEval suits teams and researchers evaluating cybersecurity risks or defensive capabilities in language models. AutoPatch users need Podman and substantial compute and storage.

Good to know, and what to watch

  • Free, MIT-licensed benchmark suite
  • Includes textual, multilingual, and visual injection tests
  • Evaluates defensive SOC capabilities
  • Supports API and self-hosted models
  • AutoPatch recommends at least 80 CPUs
  • AutoPatch recommends 8 TB storage
  • Apple Silicon MacBooks are not currently supported

CyberSecEval 4

Free

Free plan

The verdict

CyberSecEval covers several distinct security evaluations, from prompt injection and insecure code to patch generation and defensive SOC tasks. AutoPatch has substantial compute and storage requirements, so that part of the suite needs careful planning.

What the maker publishes

Automated evaluation
The original research describes automated test-case generation and an evaluation pipeline for LLM cybersecurity safety. source
AutoPatchBench
AutoPatchBench measures AI patch-generation agents on C/C++ vulnerabilities found by fuzzing. source
AutoPatch requirements
AutoPatch requires Podman and substantial compute and storage; the page recommends at least 80 CPUs and 8 TB storage and says Apple Silicon MacBooks are not currently supported. source
Code interpreter
The code interpreter benchmark assesses whether models comply with malicious prompts and categorizes judged responses as extremely malicious, potentially malicious, or non-malicious. source
Company founded
Meta’s SEC filing says the company was incorporated in Delaware in July 2004. source
Company headquarters
Meta lists its principal executive offices at 1 Meta Way, Menlo Park, California 94025. source
Company legal name
Meta’s SEC filing identifies the maker as Meta Platforms, Inc., a Delaware corporation. source
Company size
LinkedIn lists Meta as having more than 10,001 employees and headquarters in Menlo Park, California. source
Current version
CyberSecEval 4 adds AutoPatchBench for measuring an LLM agent’s ability to automatically patch vulnerabilities in native code. source
CyberSecEval 2 additions
CyberSecEval 2 added prompt-injection, code-interpreter-abuse and vulnerability-exploitation evaluations. source
CyberSecEval 3 coverage
CyberSecEval 3 evaluates eight risks across risks to third parties and risks to application developers or end users. source
CyberSecEval 3 features
CyberSecEval 3 added automated social engineering, scaled manual offensive operations and autonomous offensive operations tests. source
Documentation depth
The repository documents benchmark commands, provider formats, self-hosted integration, datasets, judging stages and developer tests. source
Extensibility
Developers can add providers by implementing an LLM class, registering it in the factory and supplying query and chat methods. source
False-refusal metric
CyberSecEval 2 introduced False Refusal Rate to quantify the tradeoff between rejecting unsafe requests and answering benign ones. source
Independent assessment
The peer-reviewed critique describes CyberSecEval as useful in a young field but identifies features that limit its utility. source
Independent cost observation
An Inspect Evals report estimated full one-epoch CyberSecEval 4 runs at roughly $29 to $213 per model depending on the model used. source
Independent implementation limit
Inspect Evals intentionally omits the autonomous-uplift and autopatching prototypes from its public CyberSecEval 4 task surface. source
Installation
The documented setup creates a Python virtual environment and installs dependencies from requirements.txt. source
Intended users
The project describes itself as tools and evaluations intended to help the community build responsibly with open generative AI models. source
License
The PurpleLlama repository states that Cybersecurity Eval benchmarks are licensed under the MIT license for research and commercial use. source
License
Cybersecurity evaluations and benchmarks in Purple Llama are licensed under MIT, while included Llama safeguard models use separate Llama Community licenses. source
Limit: narrow ruleset
The critique reports that Meta’s 89 Semgrep rules cover eight languages, versus 2,116 rules and 28 languages in the comparison repository. source
Limit: prompt construction
The critique found 23.5% of Instruct prompts problematic because they tested refusal behavior rather than genuine insecure-code propensity. source

Checked against the maker's pages on 15 Oct 2026.