Promptfoo

Promptfoo is an open-source tool for testing and securing LLM applications. You define evaluations, run them from the command line and compare results in a web viewer. Promptfoo is now part of OpenAI and remains open source.

Promptfoo runs automated evaluations of prompts and models, compares providers side by side (OpenAI, Anthropic, Azure, Bedrock, Ollama and more), and red-teams apps with vulnerability scanning. It plugs into CI/CD and can review pull requests for LLM-related security and compliance issues. It installs with npm, Homebrew or pip.

Promptfoo makes testing prompts and security-scanning LLM apps a routine part of development. It is MIT-licensed.

Stars

0

Forks

0

Commits

0

Repository

promptfoo/promptfoo

• LangSmithLangChain's platform for tracing and evaluating LLM apps.
• BraintrustPlatform for evaluating and monitoring AI products.
• OpikNo. 212Open-source LLM observability and evaluation from Comet.