Giskard
ActiveDescription
An open-source evaluation and testing library for LLM agents providing automated model scanning, bias detection, performance benchmarking, and compliance checks.
Key Features
- Scenario API for creating evaluations that test non-deterministic LLM outputs
- Built-in checks including Groundedness, Conformity, and LLM-as-judge assessments
- Red-teaming vulnerability scanner generating adversarial test suites across OWASP LLM Top-10 categories
- Multi-turn conversation testing for full agent dialog evaluation
- Modular v3 architecture with focused packages: giskard-checks, giskard-scan, and planned giskard-rag
- Async-first design with lightweight dependencies for efficient CI/CD integration
Use Cases
💡 Catch regressions after model or prompt changes with automated eval suites
💡 Validate RAG quality by checking if answers are grounded in retrieved context
💡 Red-team AI agents to identify prompt injection, data leakage, and bias vulnerabilities
💡 Generate evaluation datasets for RAG applications from knowledge bases
💡 Enforce content safety policies with conformity checks on agent outputs
Strengths & Limitations
✅ Strengths
- • Actively maintained, recent updates
- • High community interest (5.8k stars)
- • Permissive open-source license (Apache-2.0)
- • Established track record (4 years in production)
Categories
Quick Start
pip install giskard-checks && pip install giskard-scan