OpenCompass

Active

GitHub Python Apache-2.0

Description

OpenCompass is a comprehensive LLM evaluation platform supporting a wide range of models including Llama, Mistral, GPT-4, Qwen, GLM, and Claude across 100+ benchmark datasets.

Key Features

Comprehensive benchmarking across 100+ datasets covering knowledge, reasoning, math, code and more
Compatible with major LLMs — Llama, Mistral, GPT-4, Qwen, GLM, Claude, etc.
CascadeEvaluator for sequential multi-evaluator pipelines on complex assessment scenarios
Built-in GenericLLMEvaluator (LLM-as-Judge) and MATHVerifyEvaluator for math reasoning
CompassHub leaderboard and CompassRank model ranking visualizations
Recommended by Meta AI, integrated into the Llama Get Started workflow

Use Cases

💡 Run comprehensive multi-dimensional benchmarks on custom or fine-tuned models

💡 Compare different LLMs on specific tasks to support model selection decisions

💡 Track performance changes across model versions to detect capability regressions

💡 Build enterprise internal LLM evaluation pipelines for automated model acceptance testing

💡 Reproduce evaluation results from academic papers to verify claimed model capabilities

Strengths & Limitations

✅ Strengths

• Actively maintained, recent updates
• High community interest (7.4k stars)
• Permissive open-source license (Apache-2.0)
• Established track record (3 years in production)

⚠️ Limitations

• High issue backlog (394 open issues)

Quick Start

Install with pip install opencompass, configure models and datasets via OpenCompass config files, and run opencompass to start evaluation. Supports on-demand dataset loading from ModelScope, with built-in CompassHub leaderboard for result visualization.

Visit GitHub Visit Website View Docs

Related Projects

OpenAI Evals

19.4k · Python

Stale B

OpenAI's framework for evaluating LLMs and LLM systems, providing an open-source registry of benchmarks and tools for systematic model assessment.

llm-evaluationbenchmarkevals +2

· Open-source registry of evals for testing different dimensions of LLM performance
· Custom eval creation using basic and model-graded templates without writing code
· Private evals support for evaluating LLM patterns in your workflow without exposing data

View full details → View on GitHub

HELM

2.9k · Python

Active A

HELM (Holistic Evaluation of Language Models) is Stanford CRFM's open-source framework for holistic, reproducible, and transparent evaluation of foundation models including LLMs and multimodal models.

llm-evaluationbenchmarkstanford +2

· Standardized benchmarks including MMLU-Pro, GPQA, IFEval, WildBench
· Unified interface to evaluate models from OpenAI, Anthropic, Google and more
· Metrics beyond accuracy — efficiency, bias, toxicity, fairness

View on GitHub

UQLM

1.2k · Python

Active A+

CVS Health's open-source uncertainty quantification library for language models, providing UQ-based hallucination detection with confidence scoring and mitigation tools to identify and reduce unreliable LLM outputs.

hallucination-detectionuncertainty-quantificationllm-evaluation +2

· Five categories of uncertainty quantification scorers: Black-Box (consistency), White-Box (token probability), LLM-as-a-Judge, Ensemble, and Long-Text
· Black-Box scorers measure response consistency through multiple generations and comparisons, compatible with any LLM, off-the-shelf
· White-Box scorers leverage token probabilities for uncertainty estimation, single-generation scoring with minimal latency and cost

View on GitHub

Guardrails AI

7.3k · Python