Curated repos, tools, and frameworks shaping the developer ecosystem. Live data from GitHub.
by promptfoo
Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.
promptfoo is a CLI and library for evaluating and red-teaming LLM apps. Stop the trial-and-error approach - start shipping secure, reliable AI apps.
Website · Getting Started · Red Teaming · Documentation · Discord
Update (March 16, 2026): Promptfoo has joined OpenAI. Promptfoo remains open source and MIT licensed. Read the company update.
npm install -g promptfoo
promptfoo init --example getting-started
Also available via brew install promptfoo and pip install promptfoo. You can also use npx promptfoo@latest to run any command without installing.
Most LLM providers require an API key. Set yours as an environment variable:
export OPENAI_API_KEY=sk-abc123
Once you're in the example directory, run an eval and view results:
cd getting-started
promptfoo eval
promptfoo view
See Getting Started (evals) or Red Teaming (vulnerability scanning) for more.
Here's what it looks like in action:
It works on the command line too:
It also can generate security vulnerability reports:
We welcome contributions! Check out our contributing guide to get started.
Join our Discord community for help and discussion.
A set of beautifully-designed, accessible components and a code distribution platform. Works with your favorite frameworks. Open Source. Open Code.