Skip to the content
Know your tech before you buy it. Makers: get listed from $49/mo

Project Moonshot

aiverifyfoundation.sg

  • Free plan
  • Windows
  • macOS
  • Linux
  • Web
  • API
  • Self-hosted

What Project Moonshot is

Project Moonshot is a free, open-source toolkit for evaluating the safety and reliability of large language models and applications. It runs benchmark tests against open datasets and metrics, and can generate adversarial prompts with algorithmic strategies or generative models to probe potential vulnerabilities and misuse. Users can create custom evaluations from their own datasets, adding optional prompt templates, evaluation metrics, and grading scales. Results are available as interactive HTML reports and downloadable raw JSON. Moonshot can be used through a web interface, command line, library APIs, or web APIs. It supports configurable connections to providers including OpenAI, Anthropic, Together, and Hugging Face, and users can build connectors for other models or applications hosted on custom servers. The toolkit implements benchmarks recommended in IMDA’s Starter Kit for safety testing LLM-based applications. The repository identifies it as beta software under Apache License 2.0. Installation uses pip; Python 3.11 is required, and the web interface additionally requires Node.js 20.11.1 LTS or later.

Who it suits

Moonshot is intended for AI developers and compliance teams evaluating LLMs and LLM-based applications. It suits users able to install and run the toolkit in a Python environment.

Good to know, and what to watch

  • Free and open source under Apache License 2.0
  • Supports benchmark testing and adversarial probing
  • Allows custom datasets and evaluation criteria
  • Exports interactive HTML and raw JSON results
  • Identified as beta software
  • Requires Python 3.11
  • Web UI also requires Node.js 20.11.1 LTS or later

Project Moonshot

Free

Free plan

Getting started

  1. Visit the Project Moonshot website.
  2. Install Moonshot with pip, following the maker’s instructions.
  3. Use Python 3.11; install Node.js 20.11.1 LTS or later if using the Web UI.
  4. Run the Web UI locally at localhost:3000, or use the CLI or APIs.
  5. Configure a provider connection with an API key, or create a custom connector.
  6. Use Chrome for the best Web UI experience.

Limits to check

The free plan is an open-source toolkit that requires Python 3.11; its Web UI requires Node.js 20.11.1 LTS or later. The software is beta, and x86 Macs may encounter installation difficulties with moonshot-data dependencies.

The verdict

Project Moonshot provides several ways to run safety evaluations, from benchmarks to custom red-team tests. Its beta status and runtime requirements are important considerations before adoption.

Questions people ask

How much does Project Moonshot cost?

It is free. The listed Project Moonshot plan is free.

Is there a free trial?

No. The toolkit is offered as a free plan.

Which platforms does Moonshot support?

The listed platforms are API, Linux, macOS, self-hosted, web, and Windows. Its installation guide runs the Web UI locally.

Is Project Moonshot open source?

Yes. The repository identifies it as beta software under Apache License 2.0.

Which model providers can it connect to?

Documentation names OpenAI, Anthropic, Together, and Hugging Face connections using an API key. Users can also create connectors for other models or applications on custom servers.

What does it require to install?

The maker’s instructions use pip and require Python 3.11. The Web UI also requires Node.js 20.11.1 LTS or later.

What the maker publishes

Benchmarking
It runs benchmark tests on LLMs and applications using open-source datasets and metrics for performance and trust and safety risks. source
Benchmarking
Its benchmarks cover capability, quality, and trust and safety, including measures such as accuracy, bias, toxicity, and hallucination. source
Compatibility note
The installation guide recommends Chrome for the best web UI experience and says x86 Macs may encounter installation difficulties with moonshot-data dependencies. source
Custom connectors
Users can create model connectors for other models or their own LLM applications hosted on custom servers. source
Custom evaluations
Users can create recipes using their own datasets, optional prompt templates, evaluation metrics, and grading scales. source
Custom tests
Users can build benchmark tests using custom datasets, optional prompt templates, evaluation metrics, and grading scales. source
Founded
2024 source
IMDA alignment
The toolkit implements benchmarks recommended in IMDA’s Starter Kit for safety testing LLM-based applications. source
Installation
The maker's instructions install Moonshot with pip and run the web UI locally at localhost:3000. source
Integrations
Users can configure connections to their LLMs and create custom connector endpoints through the Web UI or CLI guides. source
Intended users
The project describes its users as AI developers and compliance teams evaluating LLM-based applications and LLMs. source
Intended users
The maker describes Moonshot as a tool for AI developers, compliance teams, and AI system owners evaluating LLMs and LLM applications. source
Interfaces
Moonshot can be used through a Web UI, an interactive command-line interface, library APIs, and Web APIs. source
Interfaces
Moonshot can be used through a web UI, an interactive command-line interface, library APIs, and web APIs. source
License and maturity
The repository identifies Moonshot as beta software released under the Apache Software License 2.0. source
License and status
The GitHub repository identifies the project as beta and licenses it under Apache License 2.0. source
Maker
AI Verify Foundation is a not-for-profit wholly owned subsidiary of Singapore’s Infocommunications Media Development Authority. source
Provider connections
The documentation names OpenAI, Anthropic, Together, and Hugging Face as model providers Moonshot can connect to with an API key. source
Purpose
Project Moonshot is an open-source toolkit for assessing the safety and reliability of large language models and applications through benchmark testing and red teaming. source
Purpose
Project Moonshot is an open-source toolkit for testing the safety and reliability of LLMs and LLM applications through benchmarking and red teaming. source
Red teaming
It generates adversarial prompts using algorithmic strategies or generative LLMs to probe potential vulnerabilities and misuse. source
Red teaming
The toolkit supports adversarial testing with prompt templates, context strategies, and automated attack modules. source
Reporting
It provides interactive HTML reports and downloadable raw JSON test results. source
Reports
Moonshot provides interactive HTML reports and downloadable raw JSON results for programmatic analysis. source

Checked against the maker's pages on 30 Sept 2026.