HarmBench
github.com
- Free plan
- Linux
- API
- Self-hosted
7.0 Club score Visit github.com
What the maker publishes
- Classifiers
- The project provides three classifier models for standard, contextual, and multimodal behaviors, including a validation classifier. source
- Compute requirements
- The documentation says evaluation requires enough GPUs for the target model, attack, and classifier, and its default GPU settings were chosen for 80GB A100s. source
- Customization
- Users can add Hugging Face Transformers models through the model configuration and add red-teaming methods in the baselines directory. source
- Custom methods
- Users can add red teaming methods by creating a subfolder in the baselines directory and implementing the RedTeamingMethod class. source
- Custom models
- Users can add Hugging Face Transformers models through the model configuration file, though AutoDAN, PAIR, and TAP require manual experiment configuration for new models. source
- Evaluation uses
- The framework supports evaluating red teaming methods against LLMs and evaluating LLMs against red teaming methods. source
- Evaluation workflow
- Its pipeline generates red-team test cases, prompts target models to generate completions, and classifies those completions to calculate attack success rate. source
- Execution
- The pipeline can run sequentially on the current machine, in parallel across GPUs on one machine with Ray, or with SLURM across machines. source
- Installation
- The README instructs users to clone the repository, install requirements with pip, and download the en_core_web_sm spaCy model. source
- License
- The public GitHub repository lists an MIT license. source
- Model support
- HarmBench supports Transformers-compatible LLMs, numerous closed-source APIs, and several multimodal models out of the box. source
- Pipeline
- Its evaluation pipeline generates test cases, generates model completions, and evaluates completions, with an optional test-case merging step. source
- Pipeline execution
- The pipeline can run locally, in parallel across GPUs on one machine with Ray, or through SLURM jobs. source
- Purpose
- HarmBench is an open-source framework for evaluating automated red teaming methods and LLM attacks and defenses. source
- Security and compliance
- The repository README and linked HarmBench website page provide no security or compliance claims; the website page opened here only says JavaScript must be enabled. source
- Support
- The README links to evaluation pipeline and codebase documentation for further details. source
- Support
- The repository documentation invites users to submit a pull request for support for another cluster manager. source
- Usage caveat
- The documentation says its text and multimodal classifier models are test classifiers intended only for evaluation. source
- Who it serves
- The project describes uses for evaluating red-teaming methods against LLMs and evaluating LLMs against red-teaming methods. source
Checked against the maker's pages on 8 Oct 2026.