HomeProjectsArticlesPK BoardAbout
Explore Projects

Promptfoo

Active
GitHub TypeScript MIT

Description

Test and evaluate LLM prompts, agents, and RAG pipelines. Built-in red teaming and security evaluation for reliable AI applications.

Key Features

  • Automated LLM evaluations — Batch test prompts, models, and RAG pipeline output quality
  • Red team security testing — Built-in vulnerability scanning and adversarial testing for LLM security
  • Multi-model comparison — Side-by-side comparison of OpenAI, Anthropic, Azure, Bedrock, Ollama models
  • CI/CD integration — Automate evaluation checks in continuous integration pipelines
  • Code scanning — Review LLM-related security and compliance issues in pull requests
  • Result sharing — Share evaluation results with team members for collaborative analysis

Use Cases

💡 Evaluate LLM prompt quality before production deployment
💡 Run red team security testing on AI applications to find vulnerabilities
💡 Choose the best model among multiple LLM providers
💡 Establish continuous regression testing baselines for LLM applications

Strengths & Limitations

✅ Strengths

  • • Actively maintained, recent updates
  • • High community interest (24.8k stars)
  • • Permissive open-source license (MIT)
  • • Established track record (3 years in production)

Tags

testing evaluation red-teaming prompt-testing security

Categories

📊 Observability 🛡️ Security & Guardrails

Quick Start

npm install -g promptfoo
export OPENAI_API_KEY=sk-xxx
promptfoo init --example getting-started
cd getting-started
promptfoo eval
promptfoo view
Visit GitHub

Related Projects

Giskard

5.8k · Python
Active A+

An open-source evaluation and testing library for LLM agents providing automated model scanning, bias detection, performance benchmarking, and compliance checks.

evaluationtestingllm-safety +3
  • · Scenario API for creating evaluations that test non-deterministic LLM outputs
  • · Built-in checks including Groundedness, Conformity, and LLM-as-judge assessments
  • · Red-teaming vulnerability scanner generating adversarial test suites across OWASP LLM Top-10 categories
View full details → View on GitHub

Purple Llama

4.4k · Python
Active A

Meta's set of tools to assess and improve LLM security, including safety benchmarks, prompt injection detection, and output auditing to help evaluate and enhance the safety of large language models.

securityevaluationpython +2
  • · Llama Guard models for input/output content moderation and safety filtering
  • · Prompt Guard to detect and block prompt injection and jailbreak attacks
  • · Code Shield for inference-time filtering of insecure LLM-generated code
View on GitHub

Agent Governance Toolkit

6.2k · Python
Active A

Microsoft's AI Agent Governance Toolkit providing policy enforcement, zero-trust identity, execution sandboxing, and reliability engineering for autonomous AI agents. Covers 10/10 OWASP Agentic Top 10.

securityevaluationpython +2
  • · Deterministic policy enforcement engine: intercepts and evaluates all operations before model output reaches tool calls, with YAML policy files defining allow/deny/approval-required rules — denied actions are structurally impossible
  • · Zero-trust identity and audit trail: assigns unique DIDs to each agent, generates tamper-evident audit records for every operation including active policy, request content, and decision rationale
  • · Multi-language SDK support: Python, TypeScript, .NET, Rust, and Go SDKs with unified policy evaluation APIs covering major development stacks
View full details → View on GitHub

Agentic Security

2.0k · Python
Active A

An open-source LLM vulnerability scanner and AI red teaming kit for automated security fuzzing of LLM applications, detecting jailbreaks, prompt injection, and adversarial attacks.

llm-securityred-teamingllm-fuzzer +2
  • · Multimodal attack probing across text, image, and audio inputs to test LLM robustness
  • · Multi-step jailbreak simulation with iterative attack sequences to uncover safety weaknesses
  • · Comprehensive fuzzing engine with randomized inputs to stress-test LLMs for edge cases
View on GitHub

Related Articles

安全Prompt InjectionOWASP

Agent Prompt Injection Defense: OWASP LLM01 in Practice

Based on OWASP LLM Top 10 engineering practice, this article systematically explains the seven layers of defense-in-depth for agent prompt injection: input sanitization, instruction isolation, least-privilege, output auditing, guardrails frameworks, continuous red-teaming, and kill switches -- with actionable code and toolchains.

Read more →
security-guardrailsred-teamprompt-injection

AI Agent Guardrails and Red Teaming in Practice: From Rule Engines to Adversarial Evaluation

Five-layer defense plus red-team loop, built on five open-source projects you can copy.

Read more →