promptfoo
Configurable evaluations and red-team tests for models and agent applications.
Where it stands.
A documented CLI and library for repeatable, workload-specific evaluation. The quality of assertions and test coverage determines what passing results mean.
A desk review of published evidence. We have not benchmarked or audited this software.
Behind the verdict.
Adoption signals
The repository reports usage by OpenAI and Anthropic. This is publisher-reported evidence, not an independently measured market share.
Developer experience
Declarative test configurations, a CLI, output comparisons and CI/CD integration.
Maintenance
Public repository and release links are available. Release cadence and support responsiveness were not audited.
Compatibility & access
Node-based CLI/library with multiple and custom model providers. MIT core; provider usage and hosted/enterprise features have separate terms.
What to watch for.
- A passing suite measures its configured cases rather than universal reliability.
- Model-graded assertions add judge-model cost and potential bias.
These are documented constraints or editorial implications of the architecture, not an exhaustive defect list.
Follow the evidence.
01Evaluation & CI documentation
Explains test configurations, assertions, custom providers, CLI/library and CI usage.
www.promptfoo.devRepository & reported usage
Public implementation, license and publisher-reported organizational usage.
github.comExplore alongside.
Related by catalog category.
Capabilities may differ.