Skip to the content
Know your tech before you buy it. Makers: get listed from $49/mo

Inspect AI

inspect.aisi.org.uk

  • Free plan
  • Browser extension
  • API

What Inspect AI is

Inspect AI is a free, open-source framework for evaluating large language models across coding, agentic tasks, reasoning, knowledge, behavior, and multimodal understanding. Evaluations combine datasets, solvers, and scorers, and more than 200 pre-built evaluations are available. Inspect View provides web-based monitoring and visualization of results, while the VS Code extension supports evaluation-log browsing, task running and debugging, and option configuration. The framework includes built-in support for more than 20 model providers and local inference through Hugging Face, vLLM, and SGLang. Standard tools cover web search, code execution, text editing, computer use, and web browsing, with MCP tool support. Agents from OpenAI Agents SDK, Pydantic AI, LangChain, Claude Code, Codex CLI, and Gemini CLI can be integrated. Untrusted model code can run through Docker, Kubernetes, Modal, Proxmox, Vagrant, and other systems using an extension API. Inspect is self-hosted; installation uses PyPI, and development and runs require model access.

Who it suits

Inspect AI suits teams building or assessing language-model systems that need reusable evaluations, model integrations, agent support, and safety or human-review workflows. It is aimed at technical users comfortable with self-hosted, model-dependent tooling.

Good to know, and what to watch

  • Open-source framework with a free plan
  • More than 200 pre-built evaluations
  • Supports over 20 model providers
  • VS Code evaluation-log and debugging tools
  • Multiple sandbox deployment options
  • Self-hosted deployment required
  • Evaluation runs require access to a model
  • Installation uses pip
  • Technical setup is required

Inspect AI

Free

Open source

The verdict

Inspect AI provides a broad evaluation framework with reusable components, provider integrations, agent connections, and sandboxing. Its self-hosted setup and requirement for model access make it most suitable for technical evaluation teams.

What the maker publishes

Agent integrations
Inspect can integrate agents built with OpenAI Agents SDK, Pydantic AI, and LangChain, as well as CLI agents such as Claude Code, Codex CLI, and Gemini CLI. source
Editor integration
The Inspect VS Code extension provides tools for browsing evaluation logs, running and debugging tasks, and configuring options. source
Evaluation results
Inspect View is a web-based tool for monitoring and visualizing evaluations. source
Founded
2024 source
Installation requirement
The getting-started instructions install Inspect from PyPI with pip install inspect-ai and say evaluation development and runs also require access to a model. source
Model integrations
Inspect has built-in support for more than 20 model providers and local inference with Hugging Face, vLLM, and SGLang. source
Provider examples
Documented provider groups include OpenAI, Anthropic, Google, AWS Bedrock, Azure AI, Hugging Face, vLLM, Ollama, and SGLang. source
Purpose
Inspect is an open-source framework for evaluating large language models across coding, agentic tasks, reasoning, knowledge, behavior, and multimodal understanding. source
Reusable components
Evaluations combine datasets, solvers, and scorers, and can use more than 200 pre-built evaluations. source
Sandboxing
The framework supports running untrusted model code in Docker, Kubernetes, Modal, Proxmox, Vagrant, and other systems through an extension API. source
Tools and MCP
Inspect includes standard tools for web search, code execution, text editing, computer use, and web browsing, and supports MCP tools. source

Checked against the maker's pages on 4 Oct 2026.