DeepEval
deepeval.com
- Free plan
- Open source
- Windows
- macOS
- Linux
- Web
- API
- Self-hosted
What DeepEval is
DeepEval is an open-source framework for building evaluation pipelines for AI systems. Its Pytest-native checks can run in CI/CD or as Python scripts, and its listed metrics cover areas such as hallucination, faithfulness, answer relevance, toxicity, bias and summarization. The framework evaluates text, images and audio, including conversational and voice interactions, using methods such as G-Eval, DAG, QAG and JevEval. It can create synthetic evaluation examples from a knowledge base, simulate conversations across user personas, and trace agent steps for inspection and grading. Integrations include agent frameworks such as LangChain, Pydantic AI and CrewAI, alongside model providers including OpenAI, Anthropic, Gemini and Amazon Bedrock. DeepEval is free under the listed open-source plan. The maker describes its OS offering as focused on pre-production testing, with results kept in local files and an engineer-owned test runner. Basic telemetry is sent to PostHog by default; the maker says it excludes personally identifiable information and stored results, and users can opt out.
Who it suits
DeepEval suits developers and teams building AI systems who want automated evaluation in Python scripts or CI/CD. Its stated pre-production scope and local results model may fit teams that manage testing through an engineer-owned runner.
Good to know, and what to watch
- Pytest-native evaluations run in CI/CD or Python scripts.
- Metrics cover hallucination, faithfulness, toxicity and bias.
- Evaluates text, images and audio.
- Traces agent steps for inspection and grading.
- Free open-source plan.
- OS offering is limited to pre-production testing.
- Results are kept in local files.
- Basic telemetry is on by default, though opt-out is available.
Plans
Every plan and priceEnterprise
DeepEval
Free
Starter
Free plan
Getting started
- Open the DeepEval website
- Install or run it on Linux, macOS or Windows, or use a self-hosted deployment
- Connect a supported agent framework or model provider
- Create Pytest-native evaluations and choose metrics
- Run evaluations in CI/CD or as Python scripts
Limits to check
The free DeepEval OS offering is described as limited to pre-production testing. Results are stored in local files and use an engineer-owned test runner.
The Club's review
DeepEval is a free, open-source framework for evaluating LLMs and AI systems in Python workflows. It is best for engineering teams that want repeatable pre-production checks across text, image, and audio; it is a weaker fit when evaluation needs to be shared or managed beyond the test runner.
Overview
Pytest-native evaluations can run in CI/CD or as Python scripts, giving teams a way to make model checks part of development and regression testing. Results remain in local files and the test runner is engineer-owned, which keeps the workflow close to code but leaves collaboration and centralized review to other tools. The maker describes DeepEval OS as limited to pre-production testing, so it should not be treated as a standalone production evaluation platform.
Key features
DeepEval offers more than 50 research-backed metrics, covering issues such as hallucination, faithfulness, answer relevancy, summarization, toxicity, and bias. It evaluates text, images, and audio, including conversational and voice use cases. That breadth is useful for teams testing varied AI systems with one framework, while the multiple methods—G-Eval, DAG, QAG, and JevEval—allow different evaluation approaches rather than a single fixed technique.
Synthetic goldens can be generated from a knowledge base, and conversations can be simulated across user personas. These capabilities help teams create evaluation cases without relying solely on an existing labeled set. Agent steps can also be traced, graded, and inspected in the terminal and test runner, making the framework particularly relevant to developers debugging agent behavior.
Integrations include LangChain, Pydantic AI, OpenAI Agents, LangGraph, AWS AgentCore, Strands, Google ADK, LlamaIndex, and CrewAI. Evaluation model integrations span OpenAI, Azure OpenAI, Ollama, OpenRouter, Anthropic, Amazon Bedrock, Gemini, DeepSeek, Vertex AI, Grok, Moonshot, Portkey, vLLM, LM Studio, and LiteLLM. This range helps teams work with varied stacks, though the core workflow remains developer-led.
By default, DeepEval sends basic telemetry to PostHog; it excludes personally identifiable information and stored results, and teams can opt out with DEEPEVAL_TELEMETRY_OPT_OUT=1. Data sent to Confident AI is stored in databases in its private AWS cloud, except for organizations on the VIP plan. Enterprise security features include SSO, role-based access control, granular permissions, audit logs, SOC 2 Type II, GDPR compliance, and custom data retention. Enterprise deployment is available on Confident AI Evals, either self-hosted on customer infrastructure or in the maker’s cloud, with shared workspaces, no-code evaluation workflows, and annotation queues. Prospective customers can book a demo. These enterprise capabilities address needs beyond the local open-source runner, but belong to a separate offering.
Pricing
DeepEval — free. The free plan is an Apache 2.0-licensed open-source LLM evaluation framework with a local and CI/CD test runner. It fits individual developers and teams that can manage evaluation in code and local files. There is no paid DeepEval plan described here; enterprise collaboration and deployment are offered through Confident AI Evals, with custom pricing.
Platforms
DeepEval supports Linux, macOS, and Windows, as well as self-hosted deployment. Its Python scripts and Pytest-native checks suit teams building around Python-based development workflows.
Who it's for
Choose DeepEval if engineers need a free, extensible way to run AI quality checks in development and CI/CD, especially for agent systems or multimodal applications. Look elsewhere if product, operations, or policy teams need a shared production evaluation workflow without depending on an engineer-owned runner.
Pros and cons
- Pros: A broad set of metrics and evaluation methods supports varied checks across text, images, audio, and agent behavior.
- Pros: Pytest, scripts, and CI/CD execution make evaluation repeatable within existing development workflows.
- Pros: Synthetic goldens and persona-based conversation simulation help teams build test cases, while trace inspection aids agent debugging.
- Cons: Local results and an engineer-owned runner limit DeepEval OS to pre-production workflows rather than shared production evaluation.
- Cons: Basic telemetry is enabled by default, although it excludes personally identifiable information and stored results and can be disabled.
Alternatives
For other evaluation directories, start with LLM Evaluation Tools, AI Agent Evaluation Tools, or AI LLM Evaluation Tools.
- Maxim AI is worth comparing if a freemium option with a web platform and API access suits your workflow; its free Developer plan includes up to 3 seats, one workspace, 10k logs per month, three-day data retention, and email support.
- Promptfoo is a free-plan alternative for teams that want 10k red-team probes per month, all LLM evaluation features, and local or self-hosted runs.
- Giskard offers a free open-source library with local deployment, a basic adversarial LLM vulnerability scan, and a basic RAG evaluation report.
- Whisper is a free, MIT-licensed option to install locally with Python and ffmpeg.
- Arena (formerly Chatbot Arena) offers free public leaderboard access and head-to-head battles, with a daily usage limit that resets after 24 hours.
- Braintrust has a free Starter plan with 1 GB of processed data, 10,000 scores, 14-day retention, and unlimited users, projects, and datasets.
- Galileo may suit teams seeking a paid Pro plan with 50,000 traces per month, standard RBAC, advanced analytics, and dedicated Slack support; it costs $100 a month, billed yearly.
- Parea AI offers a free plan capped at two team members, 3k logs per month with one-month retention, and 10 deployed prompts.
Verdict
DeepEval is a strong choice for Python-based engineering teams that want a free, broad evaluation toolkit wired into pre-production tests and CI/CD. Its key advantage is the range of metrics, modalities, and agent checks; its central limitation is that local, engineer-owned workflows do not provide shared production evaluation. Choose it for development gates, and look elsewhere when centralized collaboration is the priority.
The verdict
DeepEval brings a broad set of AI evaluation methods and modalities into Python-based testing workflows. Its stated limits around pre-production use and local results are important for teams considering how to manage evaluation beyond development.
Questions people ask
Is DeepEval free?
Yes. The listed DeepEval plan costs nothing and is an open-source framework under the Apache 2.0 license.
What platforms does DeepEval support?
It supports Linux, macOS, Windows and self-hosted deployment.
Can DeepEval evaluate multimodal systems?
Yes. It evaluates text, images and audio, including conversational and voice evaluations.
Which integrations are listed?
Integrations include LangChain, Pydantic AI, OpenAI Agents, LangGraph, AWS AgentCore, Strands, Google ADK, LlamaIndex and CrewAI.
Does DeepEval send telemetry?
By default it sends basic telemetry to PostHog, excluding personally identifiable information and stored results. You can opt out with DEEPEVAL_TELEMETRY_OPT_OUT=1.
Is an enterprise deployment available?
Confident AI Evals can run in the maker's cloud or be self-hosted on a customer's infrastructure.
What the maker publishes
- API and SDKs
- Confident AI exposes APIs, a TypeScript SDK and OpenTelemetry support alongside DeepEval's Python framework. source
- CLI
- The `deepeval test run` command is built on Pytest, while `evaluate()` provides a Python-programmatic interface. source
- Cloud data
- The maker says data sent to Confident AI is stored in databases in its private AWS cloud, except for organizations on the VIP plan. source
- Company relationship
- Confident AI created and maintains DeepEval and describes it as one of its open-source testing packages. source
- Company size
- LinkedIn lists Confident AI as a privately held software company with 11–50 employees and San Francisco headquarters. source
- Component evaluation
- Teams can evaluate individual spans such as tools, planners, retrievers and generators. source
- Customization
- Metrics, prompts, judge models and evaluation templates can be customized. source
- Documentation
- The documentation contains installation, quickstarts, metric references, integrations, tracing, datasets, CLI, environment-variable and troubleshooting sections. source
- Enterprise deployment
- The enterprise offering is available on Confident AI Evals and can be self-hosted on a customer's infrastructure or run in the maker's cloud. source
- Enterprise security
- The enterprise page lists SSO, role-based access control, granular permissions, audit logs, SOC 2 Type II, GDPR compliance, and custom data retention. source
- Enterprise security
- The hosted enterprise offering states SOC 2 Type II, GDPR compliance, SSO, RBAC, audit logs and custom retention. source
- Evaluation methods
- Its evaluation techniques include G-Eval, DAG, QAG, and JevEval. source
- Founders
- Y Combinator identifies Jeffrey Ip and Kritin Vongthongsri as Confident AI's founders. source
- Framework integrations
- Official integrations include LangChain, LlamaIndex, CrewAI, OpenAI Agents, LangGraph, PydanticAI, Google ADK, AgentCore, Strands and OpenTelemetry. source
- Headquarters
- San Francisco, California, United States source
- Install package
- PyPI provides a platform-independent Python wheel and source distribution for DeepEval. source
- Integrations
- Listed integrations include LangChain, Pydantic AI, OpenAI Agents, LangGraph, AWS AgentCore, Strands, Google ADK, LlamaIndex, and CrewAI. source
- License
- The repository and package identify DeepEval as Apache Software License 2.0. source
- Limit: evaluator cost
- AI Tools Atlas notes that LLM-judge metrics require provider API calls, so cost rises with dataset size and metric count. source
- Limit: reported defect
- A GitHub issue reports the introductory tutorial failing with a `ModuleNotFoundError` when running `deepeval test run`. source
- Limit: runtime
- AI Tools Atlas reports that large or multi-turn evaluations can be computationally slow and expensive. source
- Limit: team features
- AI Tools Atlas says collaboration, dataset management, monitoring and dashboards require the hosted Confident AI service. source
- Local operation
- DeepEval can run entirely locally without Confident AI or an account. source
- Local telemetry
- By default, DeepEval sends basic telemetry to PostHog, excludes personally identifiable information and stored results, and supports opting out with DEEPEVAL_TELEMETRY_OPT_OUT=1. source
Checked against the maker's pages on 15 Oct 2026.