Skip to the content
Know your tech before you buy it. Makers: get listed from $49/mo

PromptPex

microsoft.github.io

  • Windows
  • macOS
  • Linux
  • Web
  • Self-hosted

What PromptPex is

PromptPex is a free, open-source tool for developers exploring and testing AI prompts. It treats prompts as functions, extracts output rules expressed in natural language, and generates unit tests to check whether models follow those rules. It can evaluate prompt performance across multiple models, using an LLM to judge whether outputs meet specified requirements. Test generation supports scenarios, test expansions, custom instructions, and effort levels. The command-line tool accepts Markdown or text templates, Prompty files, and GitHub Models prompt YAML files. Setup materials list OpenAI, Azure OpenAI, GitHub Models, and Ollama providers. PromptPex is distributed as an npm command-line tool, can run in Docker, and has a documented local web interface. Integrations and exports include OpenAI Evals, Azure OpenAI Stored Completions, GitHub Models Evals, and GitHub Actions. Microsoft is identified as the maker. The project is intended for research, and its maker advises further evaluation before downstream use. Vague prompts may produce weaker results, and rule extraction can vary with complex or idiomatic language.

Who it suits

PromptPex is for developers who keep prompts in their codebase and want automated prompt testing across models. It suits teams able to perform additional checks for robustness, safety, harm, and bias before downstream use.

Good to know, and what to watch

  • Extracts natural-language output rules
  • Generates configurable prompt test cases
  • Evaluates prompts across models
  • Supports CLI, Docker, and local web use
  • Integrates with evaluation workflows
  • Intended for research use
  • Vague prompts can produce suboptimal results
  • Rule extraction accuracy varies
  • Requires additional downstream evaluation

The verdict

PromptPex provides open-source tooling for turning prompt requirements into tests and model evaluations. Its configuration options and documented integrations support development workflows, while the stated research status and extraction limits call for careful review.

What the maker publishes

Automatic rule extraction
PromptPex extracts output rules written in natural language in a prompt, such as a requirement to format output as JSON. source
CLI and Docker
PromptPex is packaged as an npm command-line tool and can also be run in Docker. source
Configurable generation
Test generation can be configured with scenarios, test expansions, and custom instructions. source
Deployment
The project README gives a Docker command for running PromptPex, and the setup guide describes a local web interface. source
Feedback
The transparency note invites feedback and collaboration by email and says the team will update the repository with mitigations for reported or identified issues. source
Integrations
PromptPex documents exports and integrations for OpenAI Evals, Azure OpenAI Stored Completions, GitHub Models Evals, and GitHub Actions. source
Intended audience
The project describes its intended users as developers who keep prompts in their codebase. source
Intended use
The released open-source library is intended for research, and the maker says it should not be used in downstream applications without additional evaluation of robustness, safety, harm, and bias. source
Limitations
The maker says vague prompts can produce suboptimal results and rule extraction can vary in accuracy, especially with complex or idiomatic language. source
Maker
The documentation footer identifies Microsoft as the maker of PromptPex. source
Model evaluation
PromptPex can evaluate prompt performance across models, and uses an LLM to assess whether model outputs meet specified requirements. source
Model providers
The setup guide says users can configure OpenAI, Azure OpenAI, or Ollama credentials. source
Prompt formats
The CLI supports Markdown or text templates, Prompty files, and GitHub Models prompt YAML files. source
Purpose
PromptPex is a tool for developers to explore and test AI model prompts by treating prompts as functions and generating test inputs. source
Rule extraction
It can extract output rules written in natural language in a prompt. source
Support
The transparency note invites feedback and reports of unexpected or offensive behavior by email to listed project contacts. source
Supported model providers
The CLI setup lists OpenAI, Azure OpenAI, GitHub Models, and Ollama among supported LLM providers. source
Test generation
It generates unit test cases designed to check whether a prompt makes a model follow its output rules. source
Test generation controls
Generation can be configured with scenarios, test expansions, custom instructions, and effort levels. source

Checked against the maker's pages on 8 Oct 2026.