Promptfoo

What it is

Promptfoo is an open-source tool designed to evaluate and secure large language models by testing prompts, models, agents, and RAG pipelines. It includes adversarial attacks like jailbreaks and prompt injection, making it essential for integrating security into CI/CD processes.

Promptfoo stands out as an open-source tool specifically designed to evaluate and secure large language models through adversarial attacks, making it unique among general-purpose AI testing tools.

When it fits — and when it doesn’t

Known weaknesses

Each one links to the public issue, changelog or docs page it came from, where we have one.

1. 01

   Lack of built-in regression comparison across multiple configuration files forces manual workflows

   Users with multiple test suites spread across different promptfooconfig.yaml files report that comparing current executions with previous runs is a complex manual routine.

   Source: GitHub issue #1395 ↗
2. 02

   Large install footprint makes it difficult to use as a library dependency

   The tool includes heavy dependencies like @modelcontextprotocol/sdk and @huggingface/transformers, which users report makes it hard to add promptfoo as a dependency to their own projects.

   Source: GitHub issue #7254 ↗
3. 03

   Web viewer lacks grouping for repeated tests, making reliability analysis difficult

   When using the --repeat flag to test output reliability, the web viewer displays duplicates in a flat table, making it hard to navigate and see pass/fail rates at a glance.

   Source: GitHub issue #992 ↗
4. 04

   Web viewer presents test cases in a flat table without category grouping

   Users with multiple test files referenced in a single config find the flat table presentation unclear and lack the ability to group tests into user-defined categories on a single page.

   Source: GitHub issue #1091 ↗
5. 05

   Persistence layer does not support PostgreSQL

   Users requiring alignment with organizational infrastructure recommendations report a lack of support for configuring PostgreSQL as the persistence layer.

   Source: GitHub issue #1775 ↗

What it does well

• Evaluates large language models and RAG pipelines

• Includes adversarial attacks for security testing

• Integrates with CI/CD processes

• Open-source under MIT license

Pricing

Free tier

YesOpen source — free to use

Paid from

$0

Model

Flat rate

Enterprise plan

No

Pricing not independently confirmed — check the vendor’s page.