CyberSecEval
meta-llama.github.io
- Free plan
- Open source
- macOS
- Linux
- Web
- API
- Self-hosted
What CyberSecEval is
CyberSecEval 4 is a benchmark suite for assessing cybersecurity vulnerabilities and defensive capabilities in large language models. Its evaluations cover responses to cyberattack requests, mistaken refusals of borderline benign queries, insecure code suggestions, prompt injection, and malicious instructions aimed at code interpreters. Prompt injection tests include textual, multilingual, and visual attacks, with a judge model assessing whether an injected instruction succeeded. Secure-code checks cover instruction and autocomplete settings.
The suite also includes AutoPatchBench, which measures patch-generation agents on C and C++ vulnerabilities found through fuzzing, and CyberSOCEval, developed with CrowdStrike, for malware analysis and threat-intelligence reasoning. The project supports API, Linux, and self-hosted use. Its guide lists API support for OpenAI, Anyscale, and Together, with instructions for adding custom providers for self-hosted models. Setup uses a Python virtual environment, listed requirements, and a Python command-line module. AutoPatch requires Podman and substantial compute and storage; the project recommends at least 80 CPUs and 8 TB of storage and says Apple Silicon MacBooks are not currently supported. The benchmarks use the MIT license for research and commercial use.
Who it suits
CyberSecEval suits teams and researchers evaluating cybersecurity risks or defensive capabilities in language models. AutoPatch users need Podman and substantial compute and storage.
Good to know, and what to watch
- Free, MIT-licensed benchmark suite
- Includes textual, multilingual, and visual injection tests
- Evaluates defensive SOC capabilities
- Supports API and self-hosted models
- AutoPatch recommends at least 80 CPUs
- AutoPatch recommends 8 TB storage
- Apple Silicon MacBooks are not currently supported
Plans
Every plan and priceCyberSecEval 4
Free plan
The verdict
CyberSecEval covers several distinct security evaluations, from prompt injection and insecure code to patch generation and defensive SOC tasks. AutoPatch has substantial compute and storage requirements, so that part of the suite needs careful planning.
What the maker publishes
- Automated evaluation
- The original research describes automated test-case generation and an evaluation pipeline for LLM cybersecurity safety. source
- AutoPatchBench
- AutoPatchBench measures AI patch-generation agents on C/C++ vulnerabilities found by fuzzing. source
- AutoPatch requirements
- AutoPatch requires Podman and substantial compute and storage; the page recommends at least 80 CPUs and 8 TB storage and says Apple Silicon MacBooks are not currently supported. source
- Code interpreter
- The code interpreter benchmark assesses whether models comply with malicious prompts and categorizes judged responses as extremely malicious, potentially malicious, or non-malicious. source
- Company founded
- Meta’s SEC filing says the company was incorporated in Delaware in July 2004. source
- Company headquarters
- Meta lists its principal executive offices at 1 Meta Way, Menlo Park, California 94025. source
- Company legal name
- Meta’s SEC filing identifies the maker as Meta Platforms, Inc., a Delaware corporation. source
- Company size
- LinkedIn lists Meta as having more than 10,001 employees and headquarters in Menlo Park, California. source
- Current version
- CyberSecEval 4 adds AutoPatchBench for measuring an LLM agent’s ability to automatically patch vulnerabilities in native code. source
- CyberSecEval 2 additions
- CyberSecEval 2 added prompt-injection, code-interpreter-abuse and vulnerability-exploitation evaluations. source
- CyberSecEval 3 coverage
- CyberSecEval 3 evaluates eight risks across risks to third parties and risks to application developers or end users. source
- CyberSecEval 3 features
- CyberSecEval 3 added automated social engineering, scaled manual offensive operations and autonomous offensive operations tests. source
- Documentation depth
- The repository documents benchmark commands, provider formats, self-hosted integration, datasets, judging stages and developer tests. source
- Extensibility
- Developers can add providers by implementing an LLM class, registering it in the factory and supplying query and chat methods. source
- False-refusal metric
- CyberSecEval 2 introduced False Refusal Rate to quantify the tradeoff between rejecting unsafe requests and answering benign ones. source
- Independent assessment
- The peer-reviewed critique describes CyberSecEval as useful in a young field but identifies features that limit its utility. source
- Independent cost observation
- An Inspect Evals report estimated full one-epoch CyberSecEval 4 runs at roughly $29 to $213 per model depending on the model used. source
- Independent implementation limit
- Inspect Evals intentionally omits the autonomous-uplift and autopatching prototypes from its public CyberSecEval 4 task surface. source
- Installation
- The documented setup creates a Python virtual environment and installs dependencies from requirements.txt. source
- Intended users
- The project describes itself as tools and evaluations intended to help the community build responsibly with open generative AI models. source
- License
- The PurpleLlama repository states that Cybersecurity Eval benchmarks are licensed under the MIT license for research and commercial use. source
- License
- Cybersecurity evaluations and benchmarks in Purple Llama are licensed under MIT, while included Llama safeguard models use separate Llama Community licenses. source
- Limit: narrow ruleset
- The critique reports that Meta’s 89 Semgrep rules cover eight languages, versus 2,116 rules and 28 languages in the comparison repository. source
- Limit: prompt construction
- The critique found 23.5% of Instruct prompts problematic because they tested refusal behavior rather than genuine insecure-code propensity. source
Checked against the maker's pages on 15 Oct 2026.