LM Evaluation Harness
github.com
- Free plan
- Linux
- API
- Self-hosted
What LM Evaluation Harness is
LM Evaluation Harness is a framework for evaluating generative language models across evaluation tasks. It brings together more than 60 standard academic benchmarks, with hundreds of subtasks and variants, and supports model options including Hugging Face Transformers, GPT-NeoX, Megatron-DeepSpeed, and vLLM. It can also use hosted APIs such as OpenAI and Anthropic, plus local servers compatible with OpenAI APIs. Users can define prompts and evaluation metrics, configure tasks with YAML, and extend the framework with model backends, filters, metrics, or aggregations from external packages or a local module. Publicly available prompts support reproducibility and comparison between papers, according to the project. The MIT-licensed software is free and intended for self-hosted use. The base package provides the core framework; model backends are separate optional installs. For APIs that do not provide logits or log probabilities, the documentation limits evaluation to generate-until tasks. Support is available through GitHub issues or the EleutherAI Discord.
Who it suits
This framework suits researchers, organizations, and developers evaluating language models across standard or custom tasks. It is intended for users comfortable with self-hosted software and, where needed, installing model backends separately.
Good to know, and what to watch
- Includes over 60 academic benchmarks
- Supports several model families and hosted APIs
- Custom prompts, metrics, and YAML task configuration
- Extensible with external packages or local modules
- MIT-licensed and free
- Model backends require separate optional installs
- Some APIs are limited to generate-until tasks
Plans
Every plan and priceLM Evaluation Harness
Open source
The verdict
LM Evaluation Harness offers a broad framework for running established and custom language-model evaluations. Check API limitations and backend installation needs when planning an evaluation setup.
What the maker publishes
- API integrations
- The harness supports hosted APIs including OpenAI and Anthropic, plus local servers compatible with OpenAI APIs. source
- Benchmarks
- The project lists over 60 standard academic LLM benchmarks with hundreds of subtasks and variants. source
- Custom evaluation
- Users can define custom prompts and evaluation metrics, and can configure tasks with YAML. source
- Installation
- The base package provides the core framework while model backends are installed separately as optional extras. source
- Model support
- Supported model options include Hugging Face Transformers, GPT-NeoX, Megatron-DeepSpeed, and vLLM. source
- Notable limitation
- For model APIs that do not provide logits or log probabilities, the documentation limits use to generate-until tasks. source
- Plugins
- Model backends, filters, metrics, and aggregations can be registered from external packages through entry points or loaded from a local module. source
- Purpose
- LM Evaluation Harness is a unified framework for evaluating generative language models across evaluation tasks. source
- Reproducibility
- The project says its use of publicly available prompts supports reproducibility and comparability between papers. source
- Security and license
- The repository provides the software under the MIT License, which states that it is provided “AS IS” without warranty. source
- Support
- The project directs users to open a GitHub issue or join the EleutherAI Discord for support. source
- Supported platforms
- Documented runtime options include Windows ML and ONNX Runtime providers for CPU, CUDA, DirectML, NPU, ROCm, and other accelerators. source
- Users
- The project says the harness serves as the backend for Hugging Face’s Open LLM Leaderboard and is used by research papers and organizations. source
Checked against the maker's pages on 4 Oct 2026.