HomeProjectsArticlesPK BoardAbout
Explore Projects

LM Evaluation Harness

Active
GitHub Python MIT

Description

A framework for few-shot evaluation of language models by EleutherAI, providing standardized evaluation pipelines supporting hundreds of benchmark tasks and widely adopted as a core LLM evaluation tool in the community.

Key Features

  • Over 60 standard academic benchmarks with hundreds of subtasks
  • Support for transformers, vLLM, GPT-NeoX, and multiple inference backends
  • Commercial API model evaluation for OpenAI, Anthropic, and more
  • Publicly available prompts ensure reproducibility and cross-paper comparability
  • Adapter evaluation support including LoRA via PEFT library
  • Lightweight base install with optional model backends installed separately

Use Cases

💡 LLM performance benchmark comparison
💡 Pre-release model quality assessment
💡 Model evaluation for academic papers
💡 Enterprise model selection and comparison
💡 Open LLM Leaderboard evaluation tasks

Strengths & Limitations

✅ Strengths

  • • Actively maintained, recent updates
  • • High community interest (13.9k stars)
  • • Permissive open-source license (MIT)

⚠️ Limitations

  • • High issue backlog (922 open issues)

Tags

llm-evaluation benchmark evaluation-framework language-model python

Categories

📊 Observability

Quick Start

Clone the repo and run pip install -e . for the base framework. Install the HuggingFace backend with pip install "lm_eval[hf]", then run lm_eval --model hf --model_args pretrained=EleutherAI/gpt-j-6B --tasks hellaswag --device cuda:0 to evaluate.
Visit GitHub Visit Website

Related Projects

Lighteval

2.5k · Python
Active B

HuggingFace's all-in-one toolkit for evaluating LLMs across multiple backends, deeply integrated with the HuggingFace ecosystem and providing flexible evaluation metrics and benchmark configuration.

llm-evaluationevaluation-frameworkhuggingface +2
  • · 1000+ evaluation tasks — covers knowledge, math, code, chat, multilingual, and NLU benchmarks
  • · Multi-backend support — evaluate via Accelerate, VLLM, SGLang, Nanotron, or remote inference endpoints
  • · Custom tasks and metrics — create tailored evaluation pipelines with custom task and metric definitions
View on GitHub

Inspect AI

2.7k · Python
Active A

A framework for large language model evaluations developed by the UK AI Safety Institute (AISI), providing comprehensive model capability assessment tools with support for safety and alignment testing.

llm-evaluationai-safetyevaluation-framework +2
  • · Comprehensive LLM evaluation framework created by the UK AI Safety Institute (AISI)
  • · 200+ pre-built evaluations ready to run on any model for capability assessment
  • · Built-in prompt engineering, tool usage, multi-turn dialog, and model-graded evaluations
View on GitHub

Opik

21.8k · Python
Active A+

Opik is an open-source LLM observability platform providing agent tracing, evaluation testing, and prompt experiment management to help developers monitor and optimize AI agent systems.

observabilityllm-evaluationtracing +2
  • · Comprehensive LLM observability with deep tracing of calls, conversations, and agent activity
  • · LLM-as-a-judge metrics for hallucination detection, moderation, and RAG assessment
  • · Production-ready monitoring dashboards handling 40M+ traces per day at scale
View full details → View on GitHub