Skip to the content
Know your tech before you buy it. Makers: get listed from $49/mo

HELM

crfm.stanford.edu

  • Free plan
  • Web
  • API
  • Self-hosted
7.6 Club score Visit crfm.stanford.edu

What HELM is

HELM is a free, open-source Python framework for evaluating foundation models, including large language and multimodal models. It offers datasets and standardized benchmarks such as MMLU-Pro, GPQA, IFEval, and WildBench, with a unified interface to models from providers including OpenAI, Anthropic, and Google. Users can also configure private models, such as local checkpoints or model servers on private networks, and run inference locally or through an API. Evaluation metrics cover more than accuracy, including efficiency, bias, and toxicity. A web interface supports inspection of individual prompts and responses, while leaderboards compare models and benchmarks. Official leaderboards cover Capabilities, Safety, and VHELM; evaluation domains include medicine, finance, multilinguality, and world knowledge. HELM can be installed from PyPI, used to run benchmarks and summarize results, and started as a local web server. It can also reproduce published model evaluation results. The project is self-hosted and distributed under an Apache-2.0 license. HELM entered maintenance mode on June 1, 2026, with no new features or leaderboard evaluations planned. External API changes and provider deprecations may also break scenarios or models.

Who it suits

HELM suits teams and researchers who want a self-hosted framework for reproducible model evaluation, prompt inspection, and leaderboard comparisons.

Good to know, and what to watch

  • Includes standardized benchmarks such as MMLU-Pro and GPQA
  • Supports local models and API-based inference
  • Metrics cover efficiency, bias, and toxicity
  • Web tools inspect prompts and compare leaderboard results
  • Project is in maintenance mode
  • No new features or leaderboard evaluations are planned
  • External API and model changes may break scenarios

HELM

Free

Open source

The verdict

HELM offers a broad, self-hosted evaluation workflow spanning benchmarks, metrics, and model inspection. Its maintenance status and potential compatibility breaks are important constraints for teams planning ongoing evaluations.

What the maker publishes

Benchmarks
It provides datasets and benchmarks in a standardized format, including MMLU-Pro, GPQA, IFEval and WildBench. source
Benchmarks
It provides datasets and benchmarks in a standardized format, including MMLU-Pro, GPQA, IFEval, and WildBench. source
Compatibility limit
Scenarios and models may break as external APIs change or providers deprecate models, and the maintainers recommend testing them for intended use. source
Custom models
Users can configure private models, including a checkpoint on local disk or a model server on a private network. source
Evaluation coverage
The project says it maintains leaderboards across domains such as medicine and finance, and aspects such as multilinguality and world knowledge. source
Evaluation metrics
Its metrics measure aspects beyond accuracy, including efficiency, bias and toxicity. source
Founded
2022 source
Inference options
Model deployments can run local inference or send requests to an API. source
Inspection tools
HELM includes a web UI for inspecting individual prompts and responses and a web leaderboard for comparing results. source
Inspection UI
A web UI lets users inspect individual prompts and responses. source
Install and run
The project documents installation from PyPI with `pip install crfm-helm` and commands to run benchmarks, summarize results, and start a local web server. source
Installation
The project instructs users to install HELM from PyPI with `pip install crfm-helm`. source
Known limitation
The project describes HELM as intentionally incomplete and invites the community to identify gaps and contribute scenarios, metrics and models. source
Leaderboards
A web leaderboard compares results across models and benchmarks, with official leaderboards including Capabilities, Safety, and VHELM. source
License
The repository identifies HELM as open source and links to an Apache-2.0 license. source
Maintenance
HELM entered maintenance mode on June 1, 2026; no new features or leaderboard evaluations will be added. source
Maintenance status
The repository says HELM entered maintenance mode on June 1, 2026. source
Metrics
Its metrics cover aspects beyond accuracy, including efficiency, bias, and toxicity. source
Model providers
It offers a unified interface to models from providers including OpenAI, Anthropic and Google. source
Model providers
Models from providers including OpenAI, Anthropic, and Google are accessible through a unified interface. source
Purpose
HELM is an open source Python framework for holistic, reproducible and transparent evaluation of foundation models, including large language and multimodal models. source
Purpose
HELM is an open-source Python framework for holistic, reproducible, and transparent evaluation of foundation models, including language and multimodal models. source
Reproducibility
The framework can be used to reproduce published model evaluation results. source
Support
Users are directed to open GitHub issues for questions, which maintainers address when bandwidth is available. source

Checked against the maker's pages on 8 Oct 2026.