AgentClash
agentclash.dev
- Free plan
- Web
- API
- Self-hosted
What AgentClash is
AgentClash evaluates AI agents as they work through multi-turn tasks in a real sandbox. It scores tool choices, cost, latency, recovery, and final outcomes, then can turn a failed challenge trace into a regression test for later evaluations. Each run starts in a fresh Firecracker microVM with an isolated filesystem and network, and the environment is removed afterward. YAML challenge packs define tools, policy, scoring, and starting state; agents can use file I/O, data queries, HTTP, shell, and test runners. Scoring can combine deterministic, mathematical, behavioural, and LLM-based judges with configurable weights and consensus. Provider adapters cover OpenAI, Anthropic, Gemini, xAI, Mistral, and OpenRouter, which provides access to more than 300 models. CI/CD checks can run from GitHub Actions, a webhook, or the CLI and fail builds when specified quality measures regress. AgentClash is MIT-licensed and can be self-hosted or used with its hosted backend. The Free plan includes 25 evaluation runs per month, up to four models per run, and seven-day replay retention. Pro starts at $39 a month with annual billing, and Team starts at $80 a month with annual billing.
Who it suits
AgentClash is suited to teams evaluating coding, research, SRE, multi-step operations, codebase question-answering, or support agents. It also fits developers who want agent regressions checked in CI/CD or prefer a self-hosted stack.
Good to know, and what to watch
- Scores tool choices, cost, latency, recovery, and results.
- Failed challenge traces can become permanent regression tests.
- Fresh Firecracker microVMs isolate each run.
- Works with several model providers and OpenRouter.
- MIT-licensed with self-hosted and hosted options.
- Free plan limits use to 25 eval runs monthly.
- Free plan retains replays for seven days.
- Pro annual billing starts at $39 a month.
Plans
Every plan and priceEnterprise
Free
Pro
Team
Free plan
The verdict
AgentClash links sandboxed agent evaluation with replayable failures and CI/CD regression checks. Its free tier has defined run and retention limits, while paid annual options expand capacity.
What the maker publishes
- Agent evaluation
- It evaluates multi-turn agents that take actions in a real sandbox and scores tool choices, cost, latency, recovery, and the final result. source
- Documentation
- The public documentation covers the CLI, local stack, Fleet eval sets, datasets, regression gates, multi-turn human takeover, security stress harnesses, and runtime components. source
- Integrations
- CI/CD integrations can run regression tests from GitHub Actions, a webhook, or the CLI and fail builds when correctness, cost, latency, or required evidence regresses. source
- Knowledge sources
- Knowledge sources include PDFs, wikis, Notion, codebases, and custom APIs, with provenance attached to retrieved facts. source
- Open source and hosting
- AgentClash is MIT licensed, can be self-hosted as a full stack, or used against the hosted backend; its CLI installs from npm as the agentclash package. source
- Providers
- First-class adapters support OpenAI, Anthropic, Gemini, xAI, Mistral, and OpenRouter, with more than 300 models available through OpenRouter. source
- Purpose
- AgentClash is an open-source AI-agent evaluation platform that runs agents on real tasks, scores outcomes, replays steps, and turns failures into regression tests. source
- Regression loop
- When a model fails a challenge, AgentClash freezes the failing trace into a permanent test that future evaluations replay. source
- Sandboxing
- Each agent runs in a fresh Firecracker microVM with an isolated filesystem and network, and the sandbox is torn down after the run. source
- Scoring
- Runs combine deterministic, mathematical, behavioural, and LLM-based judges with configurable consensus aggregation and weights. source
- Security
- API keys, database credentials, and OAuth tokens are stored in a scoped secret vault and injected at tool-call time without appearing in prompts, traces, or replays. source
- Tools
- Agents can use file I/O, data queries, HTTP, shell, and test runners, with declarative YAML challenge packs defining tools, policy, scoring, and starting state. source
- Workloads
- The product is positioned for coding, research, SRE, multi-step operations, codebase question answering, and support workloads. source
Checked against the maker's pages on 1 Oct 2026.