MLflow GenAI Evaluation
mlflow.org
- Free plan
- Web
- API
- Self-hosted
What MLflow GenAI Evaluation is
MLflow GenAI Evaluation helps teams evaluate LLM applications and AI agents during development and monitor them in production. Its Evaluation Datasets organize test cases, ground-truth expectations, and other evaluation data. Teams can collect feedback from end users and domain experts, attach it to traces, and retain metadata. Scoring options include LLM-as-a-Judge, custom LLM judges, and code-based scorers; automatic evaluation supports LLM judges but not code-based scorers. Results in the MLflow UI include per-record detail and comparisons between agent versions. MLflow Tracing captures latency and token usage, and lists integrations with more than 40 LLM and agent libraries and frameworks, including OpenTelemetry, LangChain, OpenAI, and Anthropic. Built-in adapters support judge models from providers including Anthropic, Bedrock, Google, and xAI. The open-source evaluation API and UI are free, with API, self-hosted, and web options. Optional basic HTTP authentication controls access to experiments, registered models, and scorers; enabling it requires a CSRF secret key and an admin password.
Who it suits
It suits teams building and operating LLM applications or AI agents that need evaluation and monitoring across the development-to-production lifecycle. Python SDK support and API access are listed.
Good to know, and what to watch
- Free open-source evaluation API and UI
- Evaluation Datasets organize test cases and expectations
- Compares per-record results across agent versions
- Tracing captures latency and token usage
- Integrations cover 40+ LLM and agent frameworks
- Automatic evaluation does not support code-based scorers
- Authentication setup requires a CSRF secret and admin password
Plans
Every plan and priceMLflow GenAI Evaluation
Free plan
The verdict
MLflow GenAI Evaluation brings datasets, scoring, feedback, and production tracing into one evaluation workflow. Teams should note the limitation on code-based scorers in automatic evaluation and configure authentication requirements if they enable it.
What the maker publishes
- Access control detail
- Basic HTTP authentication requires a configured secret key for CSRF protection and an admin password when first enabled. source
- Automatic evaluation limit
- Automatic evaluation supports LLM judges; code-based scorers are not supported in that mode. source
- Evaluation data
- Evaluation Datasets provide a centralized place to manage test cases, ground-truth expectations, and evaluation data. source
- Human feedback
- Users can collect and manage feedback from end users and domain experts, with feedback attached to traces and recorded with metadata. source
- Integrations
- MLflow Tracing lists integrations with 40+ LLM and agent libraries and frameworks, including OpenTelemetry, LangChain, OpenAI, and Anthropic. source
- Judge providers
- MLflow's evaluation quickstart says judge models can use providers including Anthropic, Bedrock, Google, and xAI through built-in adapters. source
- Judges and scorers
- MLflow includes LLM-as-a-Judge scorers and supports custom LLM judges and code-based scorers. source
- Production monitoring
- MLflow Tracing captures metrics such as latency and token usage to help monitor application performance in production. source
- Purpose
- MLflow helps teams measure, improve, and monitor the quality of LLM applications and AI agents from development through production. source
- Review and compare
- Evaluation results can be reviewed in the MLflow UI, including per-record results and comparisons across agent versions. source
- Security controls
- MLflow supports optional basic HTTP authentication for access control over experiments, registered models, and scorers. source
- Who it is for
- The product is designed for teams building and operating LLM applications and AI agents who want evaluation and monitoring across the application lifecycle. source
Checked against the maker's pages on 29 Sept 2026.