Skip to the content
Know your tech before you buy it. Makers: get listed from $49/mo

OpenAgent Eval

openagenthq.github.io

  • Free plan
  • Windows
  • macOS
  • Linux
  • API
  • Self-hosted

What the maker publishes

Data handling
The project says it runs on the user's machine without dashboards or accounts and that data never leaves the laptop. source
Embedders
Sentence Transformers is the listed built-in embedder, and custom embedders can be added through provider base classes. source
Extensibility
A plugin architecture supports custom metrics, providers, and report generators. source
Framework support
The project says it works with LangChain, LlamaIndex, and custom RAG pipelines. source
Framework support
The framework describes itself as compatible with LangChain, LlamaIndex, and custom RAG pipelines. source
Integrations
Documented LLM providers include OpenAI, Anthropic, Gemini, Groq, OpenRouter, Ollama, and a mock provider. source
License
The project is licensed under Apache License 2.0. source
LLM integrations
Documented LLM providers include OpenAI, Anthropic, Gemini, Groq, OpenRouter, Ollama, and a mock provider. source
Local operation
The project says it runs on the user's machine without telemetry or required network calls. source
Metrics
Built-in metrics cover retrieval, generation, performance, and cost, including faithfulness, latency, and token count. source
Metrics
Built-in metrics cover retrieval, generation, latency, and token cost. source
Notable dependency limit
Some retrievers and embedders require extra dependencies, with an `[all]` package extra offered for the full set. source
Notable limitation
Some retrievers and embedders require extra dependencies, such as chromadb, sentence-transformers, faiss-cpu, or qdrant-client. source
Offline use
Built-in mock providers can run a complete evaluation locally without network calls or credentials. source
Offline use
Built-in mock providers can run a complete evaluation locally without network calls or credentials. source
Purpose
OpenAgent Eval is an open-source, local-first framework for evaluating RAG systems and AI agents. source
Purpose
OpenAgent Eval is an open-source, local-first framework for evaluating RAG systems and AI agents. source
Reports
Reports can be produced in terminal, Markdown, HTML, and JSON formats, with failure analysis. source
Reports
Reports can be output in terminal, Markdown, HTML, or JSON formats, with built-in failure analysis. source
Requirements
Installation documentation lists Python 3.11 or later as a requirement. source
Requirements
The installation guide lists Python 3.11 or later as a requirement. source
Retriever integrations
Documented retrievers include Chroma, Qdrant, Pinecone, Weaviate, FAISS, PGVector, Elasticsearch, BM25, Memory, HTTP, and Mock. source
Retrievers
Documented retriever providers include Chroma, Qdrant, Pinecone, Weaviate, FAISS, PGVector, Elasticsearch, BM25, Memory, HTTP, and Mock. source
Support
The project directs users to GitHub Discussions for questions and ideas and GitHub Issues for bugs and feature requests. source

Checked against the maker's pages on 8 Oct 2026.