Promptfoo

Summary

Promptfoo is a commercial ai-security tool that tests and red-teams LLM applications, run from the command line in CI pipelines or locally by developers. It covers both evaluation, letting teams score model or prompt outputs against assertions like word-count checks or derived metrics, and adversarial testing, generating attacks through plugins such as its Telecom-specific suite or RAG Source Attribution checks, with scans configurable at the individual test-case level for prompts, providers, and structured output. It's aimed at application developers and AI security teams building on models from providers like AWS Bedrock, Azure AI Foundry, xAI, or Hugging Face via Transformers.js, all of which it can connect to directly. Launched in 2023 and still active, with 17 releases in the tracked window and the latest just 34 days old.

Promptfoo is a testing framework for evaluating and benchmarking language model prompts and applications across multiple providers.

Release history

1. snapshot-20260823 seen Aug 23, 2026 · issue 005

   Promptfoo snapshot-20260823 adds adaptive rate limiting, Transformers.js local inference, telecom red team plugins, and multi-input red team scanning.

   
   └──▷ TRY IT
   View Promptfoo runtime logs without digging through the filesystem — useful when debugging provider errors or rate limit behavior.
   copy

   $ promptfoo logs

   Cap test-case generation per strategy and label a red team run for tracking in CI.
   copy

   $ promptfoo redteam generate -d 'nightly-telecom-scan' --config redteam.yaml

   Load separate secret and environment config files in one run, and reference a secrets file path via environment variable.
   copy

   $ promptfoo eval --env-file .env --env-file .env.secrets -c promptfooconfig.yaml

   Cap strategy test-case volume and assert word count in a single test config — useful for constraining long-running red team scans.
   promptfooconfig.yaml copy

   strategies:
     - id: jailbreak
       numTests: 20
   assertions:
     - type: word-count
       threshold: 200

   • ›Adds promptfoo logs command for viewing log files directly from the CLI.
   • ›Adds -d/--description flag to redteam generate commands for labeling generated scans.
   • ›Adds --share and --no-share flags to Model Audit for controlling cloud sharing.
   • ›Adds $VAR syntax support in file paths for referencing environment variables in configuration.
   • ›Supports multiple --env-file flags to load several environment files in a single invocation.
   +21 moreshow less

   • ›Adds --extension CLI flag to load extensions via command line.
   • ›Adds numTests config key to cap the number of test cases generated per red team strategy.
   • ›Adds metadata.sessionId as a surfaced variable column in eval result tables and exports.
   • ›Adds word-count assertion type for validating response word counts.
   • ›Adds __count variable for use in derived metrics to compute averages.
   • ›Adds test-level prompts filter, providers filter, and per-test structured output configuration at the individual test case level.
   • ›Adds adaptive rate limit scheduler that automatically adjusts concurrency based on provider rate limits and response headers.
   • ›New Transformers.js provider to run Hugging Face models locally in Node.js or the browser.
   • ›New Vercel AI Gateway and Cloudflare AI Gateway providers for routing requests.
   • ›New AWS Bedrock Video provider supporting Nova Reel and Luma Ray 2 video generation.
   • ›New Azure AI Foundry Video provider for Sora video generation.
   • ›New Telecom red team plugin for industry-specific testing of telecommunications AI systems.
   • ›New RAG Source Attribution plugin to test whether RAG systems properly attribute sources in responses.
   • ›Red team scans now support multiple input variables for testing systems with complex input structures.
   • ›Adds apiBaseUrl and websocketUrl override config and function call support to the xAI Voice provider.
   • ›Adds collaboration_mode support and integrated tracing to the OpenAI Codex provider.
   • ›HTTP provider gains native session endpoint support for stateful conversations.
   • ›Code scanning adds support for fork pull requests and comment-triggered scans.
   • ›Automatic retries for transient 5xx errors in Enterprise environments.
   • ›User-rated filter in the UI to show only manually rated eval results.
   • ›Provider config hover in eval results UI to view provider configuration details inline.
2. snapshot-20260823 seen Aug 23, 2026 · issue 002

   Promptfoo snapshot-20260823 adds adaptive rate limiting, Transformers.js local inference, telecom red team plugins, and multi-input red team scanning.

   
   └──▷ TRY IT
   View Promptfoo runtime logs without digging through the filesystem — useful when debugging provider errors or rate limit behavior.
   copy

   $ promptfoo logs

   Cap test-case generation per strategy and label a red team run for tracking in CI.
   copy

   $ promptfoo redteam generate -d 'nightly-telecom-scan' --config redteam.yaml

   Load separate secret and environment config files in one run, and reference a secrets file path via environment variable.
   copy

   $ promptfoo eval --env-file .env --env-file .env.secrets -c promptfooconfig.yaml

   Cap strategy test-case volume and assert word count in a single test config — useful for constraining long-running red team scans.
   promptfooconfig.yaml copy

   strategies:
     - id: jailbreak
       numTests: 20
   assertions:
     - type: word-count
       threshold: 200

   • ›Adds promptfoo logs command for viewing log files directly from the CLI.
   • ›Adds -d/--description flag to redteam generate commands for labeling generated scans.
   • ›Adds --share and --no-share flags to Model Audit for controlling cloud sharing.
   • ›Adds $VAR syntax support in file paths for referencing environment variables in configuration.
   • ›Supports multiple --env-file flags to load several environment files in a single invocation.
   +21 moreshow less

   • ›Adds --extension CLI flag to load extensions via command line.
   • ›Adds numTests config key to cap the number of test cases generated per red team strategy.
   • ›Adds metadata.sessionId as a surfaced variable column in eval result tables and exports.
   • ›Adds word-count assertion type for validating response word counts.
   • ›Adds __count variable for use in derived metrics to compute averages.
   • ›Adds test-level prompts filter, providers filter, and per-test structured output configuration at the individual test case level.
   • ›Adds adaptive rate limit scheduler that automatically adjusts concurrency based on provider rate limits and response headers.
   • ›New Transformers.js provider to run Hugging Face models locally in Node.js or the browser.
   • ›New Vercel AI Gateway and Cloudflare AI Gateway providers for routing requests.
   • ›New AWS Bedrock Video provider supporting Nova Reel and Luma Ray 2 video generation.
   • ›New Azure AI Foundry Video provider for Sora video generation.
   • ›New Telecom red team plugin for industry-specific testing of telecommunications AI systems.
   • ›New RAG Source Attribution plugin to test whether RAG systems properly attribute sources in responses.
   • ›Red team scans now support multiple input variables for testing systems with complex input structures.
   • ›Adds apiBaseUrl and websocketUrl override config and function call support to the xAI Voice provider.
   • ›Adds collaboration_mode support and integrated tracing to the OpenAI Codex provider.
   • ›HTTP provider gains native session endpoint support for stateful conversations.
   • ›Code scanning adds support for fork pull requests and comment-triggered scans.
   • ›Automatic retries for transient 5xx errors in Enterprise environments.
   • ›User-rated filter in the UI to show only manually rated eval results.
   • ›Provider config hover in eval results UI to view provider configuration details inline.