Skip to the content
Know your tech before you buy it. Makers: get listed from $49/mo

AgentClash

agentclash.dev

  • Free plan
  • Web
  • API
  • Self-hosted
7.2 Club score Visit agentclash.dev

What AgentClash is

AgentClash evaluates AI agents as they work through multi-turn tasks in a real sandbox. It scores tool choices, cost, latency, recovery, and final outcomes, then can turn a failed challenge trace into a regression test for later evaluations. Each run starts in a fresh Firecracker microVM with an isolated filesystem and network, and the environment is removed afterward. YAML challenge packs define tools, policy, scoring, and starting state; agents can use file I/O, data queries, HTTP, shell, and test runners. Scoring can combine deterministic, mathematical, behavioural, and LLM-based judges with configurable weights and consensus. Provider adapters cover OpenAI, Anthropic, Gemini, xAI, Mistral, and OpenRouter, which provides access to more than 300 models. CI/CD checks can run from GitHub Actions, a webhook, or the CLI and fail builds when specified quality measures regress. AgentClash is MIT-licensed and can be self-hosted or used with its hosted backend. The Free plan includes 25 evaluation runs per month, up to four models per run, and seven-day replay retention. Pro starts at $39 a month with annual billing, and Team starts at $80 a month with annual billing.

Who it suits

AgentClash is suited to teams evaluating coding, research, SRE, multi-step operations, codebase question-answering, or support agents. It also fits developers who want agent regressions checked in CI/CD or prefer a self-hosted stack.

Good to know, and what to watch

  • Scores tool choices, cost, latency, recovery, and results.
  • Failed challenge traces can become permanent regression tests.
  • Fresh Firecracker microVMs isolate each run.
  • Works with several model providers and OpenRouter.
  • MIT-licensed with self-hosted and hosted options.
  • Free plan limits use to 25 eval runs monthly.
  • Free plan retains replays for seven days.
  • Pro annual billing starts at $39 a month.

Enterprise

—

Free

Free

Pro

$49/mo

Team

$100/mo

Free plan

The verdict

AgentClash links sandboxed agent evaluation with replayable failures and CI/CD regression checks. Its free tier has defined run and retention limits, while paid annual options expand capacity.

What the maker publishes

Agent evaluation
It evaluates multi-turn agents that take actions in a real sandbox and scores tool choices, cost, latency, recovery, and the final result. source
Documentation
The public documentation covers the CLI, local stack, Fleet eval sets, datasets, regression gates, multi-turn human takeover, security stress harnesses, and runtime components. source
Integrations
CI/CD integrations can run regression tests from GitHub Actions, a webhook, or the CLI and fail builds when correctness, cost, latency, or required evidence regresses. source
Knowledge sources
Knowledge sources include PDFs, wikis, Notion, codebases, and custom APIs, with provenance attached to retrieved facts. source
Open source and hosting
AgentClash is MIT licensed, can be self-hosted as a full stack, or used against the hosted backend; its CLI installs from npm as the agentclash package. source
Providers
First-class adapters support OpenAI, Anthropic, Gemini, xAI, Mistral, and OpenRouter, with more than 300 models available through OpenRouter. source
Purpose
AgentClash is an open-source AI-agent evaluation platform that runs agents on real tasks, scores outcomes, replays steps, and turns failures into regression tests. source
Regression loop
When a model fails a challenge, AgentClash freezes the failing trace into a permanent test that future evaluations replay. source
Sandboxing
Each agent runs in a fresh Firecracker microVM with an isolated filesystem and network, and the sandbox is torn down after the run. source
Scoring
Runs combine deterministic, mathematical, behavioural, and LLM-based judges with configurable consensus aggregation and weights. source
Security
API keys, database credentials, and OAuth tokens are stored in a scoped secret vault and injected at tool-call time without appearing in prompts, traces, or replays. source
Tools
Agents can use file I/O, data queries, HTTP, shell, and test runners, with declarative YAML challenge packs defining tools, policy, scoring, and starting state. source
Workloads
The product is positioned for coding, research, SRE, multi-step operations, codebase question answering, and support workloads. source

Checked against the maker's pages on 1 Oct 2026.