OpenAgent Eval
openagenthq.github.io
- Free plan
- Windows
- macOS
- Linux
- API
- Self-hosted
7.6 Club score Visit openagenthq.github.io
What the maker publishes
- Data handling
- The project says it runs on the user's machine without dashboards or accounts and that data never leaves the laptop. source
- Embedders
- Sentence Transformers is the listed built-in embedder, and custom embedders can be added through provider base classes. source
- Extensibility
- A plugin architecture supports custom metrics, providers, and report generators. source
- Framework support
- The project says it works with LangChain, LlamaIndex, and custom RAG pipelines. source
- Framework support
- The framework describes itself as compatible with LangChain, LlamaIndex, and custom RAG pipelines. source
- Integrations
- Documented LLM providers include OpenAI, Anthropic, Gemini, Groq, OpenRouter, Ollama, and a mock provider. source
- License
- The project is licensed under Apache License 2.0. source
- LLM integrations
- Documented LLM providers include OpenAI, Anthropic, Gemini, Groq, OpenRouter, Ollama, and a mock provider. source
- Local operation
- The project says it runs on the user's machine without telemetry or required network calls. source
- Metrics
- Built-in metrics cover retrieval, generation, performance, and cost, including faithfulness, latency, and token count. source
- Metrics
- Built-in metrics cover retrieval, generation, latency, and token cost. source
- Notable dependency limit
- Some retrievers and embedders require extra dependencies, with an `[all]` package extra offered for the full set. source
- Notable limitation
- Some retrievers and embedders require extra dependencies, such as chromadb, sentence-transformers, faiss-cpu, or qdrant-client. source
- Offline use
- Built-in mock providers can run a complete evaluation locally without network calls or credentials. source
- Offline use
- Built-in mock providers can run a complete evaluation locally without network calls or credentials. source
- Purpose
- OpenAgent Eval is an open-source, local-first framework for evaluating RAG systems and AI agents. source
- Purpose
- OpenAgent Eval is an open-source, local-first framework for evaluating RAG systems and AI agents. source
- Reports
- Reports can be produced in terminal, Markdown, HTML, and JSON formats, with failure analysis. source
- Reports
- Reports can be output in terminal, Markdown, HTML, or JSON formats, with built-in failure analysis. source
- Requirements
- Installation documentation lists Python 3.11 or later as a requirement. source
- Requirements
- The installation guide lists Python 3.11 or later as a requirement. source
- Retriever integrations
- Documented retrievers include Chroma, Qdrant, Pinecone, Weaviate, FAISS, PGVector, Elasticsearch, BM25, Memory, HTTP, and Mock. source
- Retrievers
- Documented retriever providers include Chroma, Qdrant, Pinecone, Weaviate, FAISS, PGVector, Elasticsearch, BM25, Memory, HTTP, and Mock. source
- Support
- The project directs users to GitHub Discussions for questions and ideas and GitHub Issues for bugs and feature requests. source
Checked against the maker's pages on 8 Oct 2026.