Skip to the content
Know your tech before you buy it. Makers: get listed from $49/mo

HarmBench

github.com

  • Free plan
  • Linux
  • API
  • Self-hosted
7.0 Club score Visit github.com

What the maker publishes

Classifiers
The project provides three classifier models for standard, contextual, and multimodal behaviors, including a validation classifier. source
Compute requirements
The documentation says evaluation requires enough GPUs for the target model, attack, and classifier, and its default GPU settings were chosen for 80GB A100s. source
Customization
Users can add Hugging Face Transformers models through the model configuration and add red-teaming methods in the baselines directory. source
Custom methods
Users can add red teaming methods by creating a subfolder in the baselines directory and implementing the RedTeamingMethod class. source
Custom models
Users can add Hugging Face Transformers models through the model configuration file, though AutoDAN, PAIR, and TAP require manual experiment configuration for new models. source
Evaluation uses
The framework supports evaluating red teaming methods against LLMs and evaluating LLMs against red teaming methods. source
Evaluation workflow
Its pipeline generates red-team test cases, prompts target models to generate completions, and classifies those completions to calculate attack success rate. source
Execution
The pipeline can run sequentially on the current machine, in parallel across GPUs on one machine with Ray, or with SLURM across machines. source
Installation
The README instructs users to clone the repository, install requirements with pip, and download the en_core_web_sm spaCy model. source
License
The public GitHub repository lists an MIT license. source
Model support
HarmBench supports Transformers-compatible LLMs, numerous closed-source APIs, and several multimodal models out of the box. source
Pipeline
Its evaluation pipeline generates test cases, generates model completions, and evaluates completions, with an optional test-case merging step. source
Pipeline execution
The pipeline can run locally, in parallel across GPUs on one machine with Ray, or through SLURM jobs. source
Purpose
HarmBench is an open-source framework for evaluating automated red teaming methods and LLM attacks and defenses. source
Security and compliance
The repository README and linked HarmBench website page provide no security or compliance claims; the website page opened here only says JavaScript must be enabled. source
Support
The README links to evaluation pipeline and codebase documentation for further details. source
Support
The repository documentation invites users to submit a pull request for support for another cluster manager. source
Usage caveat
The documentation says its text and multimodal classifier models are test classifiers intended only for evaluation. source
Who it serves
The project describes uses for evaluating red-teaming methods against LLMs and evaluating LLMs against red-teaming methods. source

Checked against the maker's pages on 8 Oct 2026.