PromptPex
microsoft.github.io
- Windows
- macOS
- Linux
- Web
- Self-hosted
What PromptPex is
PromptPex is a free, open-source tool for developers exploring and testing AI prompts. It treats prompts as functions, extracts output rules expressed in natural language, and generates unit tests to check whether models follow those rules. It can evaluate prompt performance across multiple models, using an LLM to judge whether outputs meet specified requirements. Test generation supports scenarios, test expansions, custom instructions, and effort levels. The command-line tool accepts Markdown or text templates, Prompty files, and GitHub Models prompt YAML files. Setup materials list OpenAI, Azure OpenAI, GitHub Models, and Ollama providers. PromptPex is distributed as an npm command-line tool, can run in Docker, and has a documented local web interface. Integrations and exports include OpenAI Evals, Azure OpenAI Stored Completions, GitHub Models Evals, and GitHub Actions. Microsoft is identified as the maker. The project is intended for research, and its maker advises further evaluation before downstream use. Vague prompts may produce weaker results, and rule extraction can vary with complex or idiomatic language.
Who it suits
PromptPex is for developers who keep prompts in their codebase and want automated prompt testing across models. It suits teams able to perform additional checks for robustness, safety, harm, and bias before downstream use.
Good to know, and what to watch
- Extracts natural-language output rules
- Generates configurable prompt test cases
- Evaluates prompts across models
- Supports CLI, Docker, and local web use
- Integrates with evaluation workflows
- Intended for research use
- Vague prompts can produce suboptimal results
- Rule extraction accuracy varies
- Requires additional downstream evaluation
The verdict
PromptPex provides open-source tooling for turning prompt requirements into tests and model evaluations. Its configuration options and documented integrations support development workflows, while the stated research status and extraction limits call for careful review.
What the maker publishes
- Automatic rule extraction
- PromptPex extracts output rules written in natural language in a prompt, such as a requirement to format output as JSON. source
- CLI and Docker
- PromptPex is packaged as an npm command-line tool and can also be run in Docker. source
- Configurable generation
- Test generation can be configured with scenarios, test expansions, and custom instructions. source
- Deployment
- The project README gives a Docker command for running PromptPex, and the setup guide describes a local web interface. source
- Feedback
- The transparency note invites feedback and collaboration by email and says the team will update the repository with mitigations for reported or identified issues. source
- Integrations
- PromptPex documents exports and integrations for OpenAI Evals, Azure OpenAI Stored Completions, GitHub Models Evals, and GitHub Actions. source
- Intended audience
- The project describes its intended users as developers who keep prompts in their codebase. source
- Intended use
- The released open-source library is intended for research, and the maker says it should not be used in downstream applications without additional evaluation of robustness, safety, harm, and bias. source
- Limitations
- The maker says vague prompts can produce suboptimal results and rule extraction can vary in accuracy, especially with complex or idiomatic language. source
- Maker
- The documentation footer identifies Microsoft as the maker of PromptPex. source
- Model evaluation
- PromptPex can evaluate prompt performance across models, and uses an LLM to assess whether model outputs meet specified requirements. source
- Model providers
- The setup guide says users can configure OpenAI, Azure OpenAI, or Ollama credentials. source
- Prompt formats
- The CLI supports Markdown or text templates, Prompty files, and GitHub Models prompt YAML files. source
- Purpose
- PromptPex is a tool for developers to explore and test AI model prompts by treating prompts as functions and generating test inputs. source
- Rule extraction
- It can extract output rules written in natural language in a prompt. source
- Support
- The transparency note invites feedback and reports of unexpected or offensive behavior by email to listed project contacts. source
- Supported model providers
- The CLI setup lists OpenAI, Azure OpenAI, GitHub Models, and Ollama among supported LLM providers. source
- Test generation
- It generates unit test cases designed to check whether a prompt makes a model follow its output rules. source
- Test generation controls
- Generation can be configured with scenarios, test expansions, custom instructions, and effort levels. source
Checked against the maker's pages on 8 Oct 2026.