Skip to the content
Know your tech before you buy it. Makers: get listed from $49/mo

LM Evaluation Harness

github.com

  • Free plan
  • Linux
  • API
  • Self-hosted
6.9 Club score Visit github.com

What LM Evaluation Harness is

LM Evaluation Harness is a framework for evaluating generative language models across evaluation tasks. It brings together more than 60 standard academic benchmarks, with hundreds of subtasks and variants, and supports model options including Hugging Face Transformers, GPT-NeoX, Megatron-DeepSpeed, and vLLM. It can also use hosted APIs such as OpenAI and Anthropic, plus local servers compatible with OpenAI APIs. Users can define prompts and evaluation metrics, configure tasks with YAML, and extend the framework with model backends, filters, metrics, or aggregations from external packages or a local module. Publicly available prompts support reproducibility and comparison between papers, according to the project. The MIT-licensed software is free and intended for self-hosted use. The base package provides the core framework; model backends are separate optional installs. For APIs that do not provide logits or log probabilities, the documentation limits evaluation to generate-until tasks. Support is available through GitHub issues or the EleutherAI Discord.

Who it suits

This framework suits researchers, organizations, and developers evaluating language models across standard or custom tasks. It is intended for users comfortable with self-hosted software and, where needed, installing model backends separately.

Good to know, and what to watch

  • Includes over 60 academic benchmarks
  • Supports several model families and hosted APIs
  • Custom prompts, metrics, and YAML task configuration
  • Extensible with external packages or local modules
  • MIT-licensed and free
  • Model backends require separate optional installs
  • Some APIs are limited to generate-until tasks

LM Evaluation Harness

Free

Open source

The verdict

LM Evaluation Harness offers a broad framework for running established and custom language-model evaluations. Check API limitations and backend installation needs when planning an evaluation setup.

What the maker publishes

API integrations
The harness supports hosted APIs including OpenAI and Anthropic, plus local servers compatible with OpenAI APIs. source
Benchmarks
The project lists over 60 standard academic LLM benchmarks with hundreds of subtasks and variants. source
Custom evaluation
Users can define custom prompts and evaluation metrics, and can configure tasks with YAML. source
Installation
The base package provides the core framework while model backends are installed separately as optional extras. source
Model support
Supported model options include Hugging Face Transformers, GPT-NeoX, Megatron-DeepSpeed, and vLLM. source
Notable limitation
For model APIs that do not provide logits or log probabilities, the documentation limits use to generate-until tasks. source
Plugins
Model backends, filters, metrics, and aggregations can be registered from external packages through entry points or loaded from a local module. source
Purpose
LM Evaluation Harness is a unified framework for evaluating generative language models across evaluation tasks. source
Reproducibility
The project says its use of publicly available prompts supports reproducibility and comparability between papers. source
Security and license
The repository provides the software under the MIT License, which states that it is provided “AS IS” without warranty. source
Support
The project directs users to open a GitHub issue or join the EleutherAI Discord for support. source
Supported platforms
Documented runtime options include Windows ML and ONNX Runtime providers for CPU, CUDA, DirectML, NPU, ROCm, and other accelerators. source
Users
The project says the harness serves as the backend for Hugging Face’s Open LLM Leaderboard and is used by research papers and organizations. source

Checked against the maker's pages on 4 Oct 2026.