Promptfoo is an open-source CLI and library for evaluating and red-teaming LLM-based applications and agents.
Follow Promptfoo to get an email when its listing changes, such as new pricing or features. One digest a day at most, and you can unsubscribe anytime.
You declare test cases and assertions, compare prompts and models side by side, run automated red teaming and vulnerability scans against agents and RAG pipelines, and wire the checks into CI/CD and pull request code scanning.
- 01
Declarative evals with assertions, run via `promptfoo eval` and inspected in a local web viewer (`promptfoo view`)
- 02
Side-by-side model comparison across OpenAI, Anthropic, Azure, Bedrock, Ollama and many other providers
- 03
Red teaming and vulnerability scanning for LLM apps, agents and RAG pipelines
- 04
CI/CD integration for automated regression checks
- 05
Code scanning that reviews pull requests for LLM-related security and compliance issues
- 06
Install with npm, Homebrew or pip, or run without installing via `npx promptfoo@latest`
Open Source
Free and open source under the MIT license; install via npm, Homebrew or pip, or run with `npx promptfoo@latest`. You pay your own model provider API costs. Promptfoo joined OpenAI in March 2026 and states it remains open source and MIT-licensed; enterprise offerings are not priced in the repository.
Best for
Developers and AI teams shipping LLM features or agents who want repeatable evals, model comparisons and security red teaming in local runs and CI.
Not ideal for
People looking for a coding agent or test generator for ordinary application code; Promptfoo tests LLM behaviour and requires writing eval configs and supplying provider API keys.
The most widely adopted open-source toolkit for LLM evals and red teaming, actively released and now backed by OpenAI while staying MIT-licensed; a default choice for testing prompts and agents in CI.
Released 0.125.0 on October 23, 2026, adding new assertion types (ROUGE-L/ROUGE-S, trace span filters), eval safe mode and stability reporting, and making the Claude Agent and Codex SDK providers opt-in; the repository has 26K+ GitHub stars and appeared on GitHub Trending (TypeScript) on October 23, 2026. Promptfoo announced it was joining OpenAI on March 9, 2026.
As of
Is Promptfoo free?+
Yes - Promptfoo is Open Source. Free and open source under the MIT license; install via npm, Homebrew or pip, or run with `npx promptfoo@latest`. You pay your own model provider API costs. Promptfoo joined OpenAI in March 2026 and states it remains open source and MIT-licensed; enterprise offerings are not priced in the repository.
Is Promptfoo open source?+
Yes - Promptfoo is open source. Free and open source under the MIT license; install via npm, Homebrew or pip, or run with `npx promptfoo@latest`. You pay your own model provider API costs. Promptfoo joined OpenAI in March 2026 and states it remains open source and MIT-licensed; enterprise offerings are not priced in the repository.
Who is Promptfoo best for?+
Developers and AI teams shipping LLM features or agents who want repeatable evals, model comparisons and security red teaming in local runs and CI.
Who is Promptfoo not ideal for?+
People looking for a coding agent or test generator for ordinary application code; Promptfoo tests LLM behaviour and requires writing eval configs and supplying provider API keys.
What are the best Promptfoo alternatives?+
The closest Promptfoo alternatives on ai.dosa.dev are GitAuto, AgentDiff, agent-qa - all listed under AI Testing & Quality Assurance.
Who makes Promptfoo?+
Promptfoo is developed by Promptfoo (part of OpenAI). It is listed in the AI Testing & Quality Assurance category on ai.dosa.dev.
Favorite this tool to revisit it later, or Zap it to contribute to the public vote count.