LM Evaluation Harness

5.2easy start · #28 of 30
in AI LLM Evaluation Tools

Free to practise onnot on record
Free trialnot on record
Well documentednot on record
Runs where you worknot on record

Runs on api, Linux, self-hosted.

LM Evaluation Harness is ranked #28 of 30 in AI LLM evaluation tools on The Geeks Club. It runs on API, Linux, Self-hosted. There is a free plan.

LM Evaluation Harness plans and pricing

All plans

LM Evaluation Harness Free MIT-licensed software · model backends and task dependencies may require separate optional installs github.com · 4 Oct 2026

Compared on AI LLM evaluation tools

Deployment: self-hosted

Best LM Evaluation Harness alternatives

See all 12

Weights & Biases 0 of 4 in the starter kit Lesson 18.0 Evidently AI 0 of 4 in the starter kit Lesson 27.9 Opik 0 of 4 in the starter kit Lesson 37.7 Maxim AI 0 of 4 in the starter kit Lesson 47.6 Vellum 0 of 4 in the starter kit Lesson 57.4 Promptfoo 0 of 4 in the starter kit Lesson 67.3

The record

Purpose: LM Evaluation Harness is a unified framework for evaluating generative language models across evaluation tasks.
Benchmarks: The project lists over 60 standard academic LLM benchmarks with hundreds of subtasks and variants.
Model support: Supported model options include Hugging Face Transformers, GPT-NeoX, Megatron-DeepSpeed, and vLLM.
API integrations: The harness supports hosted APIs including OpenAI and Anthropic, plus local servers compatible with OpenAI APIs.
Custom evaluation: Users can define custom prompts and evaluation metrics, and can configure tasks with YAML.
Plugins: Model backends, filters, metrics, and aggregations can be registered from external packages through entry points or loaded from a local module.
Reproducibility: The project says its use of publicly available prompts supports reproducibility and comparability between papers.
Installation: The base package provides the core framework while model backends are installed separately as optional extras.
Supported platforms: Documented runtime options include Windows ML and ONNX Runtime providers for CPU, CUDA, DirectML, NPU, ROCm, and other accelerators.
Security and license: The repository provides the software under the MIT License, which states that it is provided “AS IS” without warranty.
Support: The project directs users to open a GitHub issue or join the EleutherAI Discord for support.
Notable limitation: For model APIs that do not provide logits or log probabilities, the documentation limits use to generate-until tasks.
Users: The project says the harness serves as the backend for Hugging Face’s Open LLM Leaderboard and is used by research papers and organizations.
Price from: Free

Where it ranks on The Geeks Club

Best AI LLM Evaluation Tools in 2026#28 of 30
Best LLM Evaluation Tools in 2026#25 of 29

Is LM Evaluation Harness yours?

Claim it for free: prove the domain, then correct facts, plans and screenshots. An editor reviews every change.

Claim it free Promote LM Evaluation Harness

Sources

github.com/EleutherAI/lm-evaluation-harness· checked 4 Oct 2026
github.com/EleutherAI/lm-evaluation-harness/blob/m· checked 4 Oct 2026