HELM
crfm.stanford.edu
- Free plan
- Web
- API
- Self-hosted
What HELM is
HELM is a free, open-source Python framework for evaluating foundation models, including large language and multimodal models. It offers datasets and standardized benchmarks such as MMLU-Pro, GPQA, IFEval, and WildBench, with a unified interface to models from providers including OpenAI, Anthropic, and Google. Users can also configure private models, such as local checkpoints or model servers on private networks, and run inference locally or through an API. Evaluation metrics cover more than accuracy, including efficiency, bias, and toxicity. A web interface supports inspection of individual prompts and responses, while leaderboards compare models and benchmarks. Official leaderboards cover Capabilities, Safety, and VHELM; evaluation domains include medicine, finance, multilinguality, and world knowledge. HELM can be installed from PyPI, used to run benchmarks and summarize results, and started as a local web server. It can also reproduce published model evaluation results. The project is self-hosted and distributed under an Apache-2.0 license. HELM entered maintenance mode on June 1, 2026, with no new features or leaderboard evaluations planned. External API changes and provider deprecations may also break scenarios or models.
Who it suits
HELM suits teams and researchers who want a self-hosted framework for reproducible model evaluation, prompt inspection, and leaderboard comparisons.
Good to know, and what to watch
- Includes standardized benchmarks such as MMLU-Pro and GPQA
- Supports local models and API-based inference
- Metrics cover efficiency, bias, and toxicity
- Web tools inspect prompts and compare leaderboard results
- Project is in maintenance mode
- No new features or leaderboard evaluations are planned
- External API and model changes may break scenarios
Plans
Every plan and priceHELM
Open source
The verdict
HELM offers a broad, self-hosted evaluation workflow spanning benchmarks, metrics, and model inspection. Its maintenance status and potential compatibility breaks are important constraints for teams planning ongoing evaluations.
What the maker publishes
- Benchmarks
- It provides datasets and benchmarks in a standardized format, including MMLU-Pro, GPQA, IFEval and WildBench. source
- Benchmarks
- It provides datasets and benchmarks in a standardized format, including MMLU-Pro, GPQA, IFEval, and WildBench. source
- Compatibility limit
- Scenarios and models may break as external APIs change or providers deprecate models, and the maintainers recommend testing them for intended use. source
- Custom models
- Users can configure private models, including a checkpoint on local disk or a model server on a private network. source
- Evaluation coverage
- The project says it maintains leaderboards across domains such as medicine and finance, and aspects such as multilinguality and world knowledge. source
- Evaluation metrics
- Its metrics measure aspects beyond accuracy, including efficiency, bias and toxicity. source
- Founded
- 2022 source
- Inference options
- Model deployments can run local inference or send requests to an API. source
- Inspection tools
- HELM includes a web UI for inspecting individual prompts and responses and a web leaderboard for comparing results. source
- Inspection UI
- A web UI lets users inspect individual prompts and responses. source
- Install and run
- The project documents installation from PyPI with `pip install crfm-helm` and commands to run benchmarks, summarize results, and start a local web server. source
- Installation
- The project instructs users to install HELM from PyPI with `pip install crfm-helm`. source
- Known limitation
- The project describes HELM as intentionally incomplete and invites the community to identify gaps and contribute scenarios, metrics and models. source
- Leaderboards
- A web leaderboard compares results across models and benchmarks, with official leaderboards including Capabilities, Safety, and VHELM. source
- License
- The repository identifies HELM as open source and links to an Apache-2.0 license. source
- Maintenance
- HELM entered maintenance mode on June 1, 2026; no new features or leaderboard evaluations will be added. source
- Maintenance status
- The repository says HELM entered maintenance mode on June 1, 2026. source
- Metrics
- Its metrics cover aspects beyond accuracy, including efficiency, bias, and toxicity. source
- Model providers
- It offers a unified interface to models from providers including OpenAI, Anthropic and Google. source
- Model providers
- Models from providers including OpenAI, Anthropic, and Google are accessible through a unified interface. source
- Purpose
- HELM is an open source Python framework for holistic, reproducible and transparent evaluation of foundation models, including large language and multimodal models. source
- Purpose
- HELM is an open-source Python framework for holistic, reproducible, and transparent evaluation of foundation models, including language and multimodal models. source
- Reproducibility
- The framework can be used to reproduce published model evaluation results. source
- Support
- Users are directed to open GitHub issues for questions, which maintainers address when bandwidth is available. source
Checked against the maker's pages on 8 Oct 2026.