Strands Evals
strandsagents.com
- Free plan
- Windows
- macOS
- Linux
- API
- Self-hosted
What Strands Evals is
Strands Evals is a free, open-source Python SDK and CLI for evaluating AI agent behavior during development and after deployment. It scores outputs and trajectories, helps diagnose failures, probes unsafe behavior, and can simulate users and tools. Evaluations can cover an individual output, tool call, trace, or full session, with multiple evaluators combined in one experiment. Built-in groups address quality, safety, multimodal responses, agent behavior, and skill selection and instruction following. Code-based checks can run without an LLM judge, and developers can extend the Evaluator base class for their own logic. Install the package with pip as strands-agents-evals; the strands-evals CLI can generate experiments, validate JSON, run evaluations, create reports, and diagnose sessions. It can retrieve traces from AWS CloudWatch Logs, Langfuse, and OpenSearch, and evaluate production or staging traces without rerunning agents. The quickstart uses Amazon Bedrock with Claude as its default judge and requires credentials authorized to invoke Claude. Red-teaming is experimental and may change in a minor release.
Who it suits
It suits developers who want to validate agent behavior, measure changes, and evaluate agents during development cycles. It can also help teams examine production or staging traces without rerunning the agents.
Good to know, and what to watch
- Free, open-source Python SDK and CLI.
- Checks outputs, tool calls, traces, and sessions.
- Deterministic checks can run without an LLM judge.
- Can evaluate existing production or staging traces.
- CLI supports validation and evaluation in CI workflows.
- Quickstart example requires authorized Claude credentials.
- Red-teaming API is experimental and may change.
- Some trace integrations may require optional package extras.
Plans
Every plan and priceStrands Evals SDK
Open source
The verdict
Strands Evals provides multiple evaluation levels, built-in checks, and CLI support for repeatable evaluation workflows. Account for the default judge credentials and the experimental status of its red-teaming API.
What the maker publishes
- Built-in evaluators
- Built-in evaluator groups cover quality, safety, multimodal responses, agentic behavior, and skill selection and instruction following. source
- Custom evaluators
- Developers can implement domain-specific evaluation logic by extending the base Evaluator class. source
- Default judge
- The quickstart states that evaluators use Amazon Bedrock with Claude as the default judge model, and that running its example requires credentials authorized to invoke Claude. source
- Deterministic checks
- Deterministic evaluators run code-based checks without an LLM judge and include Equals, Contains, StartsWith, ToolCalled, StateEquals, and SkillInvoked. source
- Evaluation levels
- Evaluators can assess a single output, tool call, trace, or full session, and multiple evaluators can be combined in one experiment. source
- Experimental feature
- The red-teaming API is marked experimental and may change in a minor release. source
- Intended users
- The SDK is presented for developers who want to validate agent behavior, measure improvements, and evaluate agents during development cycles. source
- License
- The maker’s announcement identifies Strands Agents as an open-source project licensed under Apache License 2.0. source
- Local and remote use
- The SDK can run evaluations against production or staging traces without rerunning the agents that produced them. source
- Purpose
- Strands Evals measures agent behavior before shipping and after deployment by scoring outputs and trajectories, diagnosing failures, probing unsafe behavior, and simulating users and tools. source
- SDK and CLI
- The package installs with pip as strands-agents-evals and provides both a Python API and the strands-evals command-line interface. source
- Test generation and CI
- The CLI can generate experiments, validate experiment JSON, run evaluations, render reports, and diagnose sessions; its documentation shows validation and evaluation steps in a CI workflow. source
- Trace integrations
- Trace providers fetch data from AWS CloudWatch Logs, Langfuse, and OpenSearch, with optional package extras for Langfuse and OpenSearch. source
- Trace mapping
- Session mappers include support for Strands in-memory spans, LangChain OpenTelemetry spans, and OpenInference instrumentation such as Arize Phoenix. source
Checked against the maker's pages on 4 Oct 2026.