HomeProjectsArticlesPK BoardAbout
Explore Projects

AutoRAG

Active
GitHub TypeScript NOASSERTION

Description

AutoRAG is an open-source RAG evaluation and optimization framework using AutoML-style automation to help developers automatically find the best RAG pipeline configurations and benchmark them.

Key Features

  • AutoML-style automation to automatically find the optimal RAG pipeline for your data
  • End-to-end data creation: parsing, chunking, and QA dataset generation
  • Supports multiple parsing modules (pdfminer, etc.) and chunking strategies
  • Dashboard for visualizing and comparing RAG pipeline evaluation results
  • HuggingFace Spaces integration with pre-built chatbot and optimization demos
  • Custom LLM and embedding model support for evaluation

Use Cases

💡 Automatically optimizing RAG pipeline configuration for specific datasets
💡 Evaluating and benchmarking different retrieval and generation modules
💡 Creating QA training datasets from raw documents for RAG evaluation
💡 Comparing RAG module combinations to find the best performing setup
💡 Building production-ready RAG systems with data-driven configuration

Strengths & Limitations

✅ Strengths

  • • Actively maintained, recent updates
  • • High community interest (5.1k stars)
  • • Established track record (2 years in production)

⚠️ Limitations

  • • No clear open-source license

Tags

rag evaluation optimization automl python

Categories

📚 RAG Tools

Quick Start

```bash
pip install AutoRAG

# Create QA dataset
python -c "
from autorag.parser import Parser
parser = Parser(data_path_glob='data/*')
parser.start_parsing('parse_config.yaml')
"

# Run optimization
autorag run --yaml rag_config.yaml --qa_path qa.parquet --corpus_path corpus.parquet

# Launch dashboard
autorag dashboard --project_path ./projects/[project_name]
```
Visit GitHub Visit Website View Docs

Related Projects

PromptTools

3.1k · Python
Stale B

PromptTools provides open-source tools for prompt testing and experimentation, supporting multiple LLMs (OpenAI, LLaMA) and vector databases (Chroma, Weaviate, LanceDB) to help developers systematically evaluate and optimize RAG systems.

prompt-testingragevaluation +3
  • · Open-source prompt testing and experimentation across multiple LLM providers
  • · Supports OpenAI, Anthropic, LLaMA, Mistral, Gemini, HuggingFace, and more
  • · Vector database evaluation: Chroma, Weaviate, Qdrant, LanceDB, Pinecone
View on GitHub

RAGatouille

4.0k · Python
Stale C

Easily use and train state of the art late-interaction retrieval methods (ColBERT) in any RAG pipeline. Designed for modularity and ease-of-use, backed by research.

ragpythonembedding +1
  • · ColBERT late-interaction retrieval: wraps advanced ColBERT models into simple APIs, enabling late-interaction retrieval methods in RAG pipelines without deep IR research knowledge
  • · End-to-end training and fine-tuning: built-in RAGTrainer and TrainingDataProcessor with automatic deduplication, positive/negative pairing and hard negative mining, supporting training from scratch or fine-tuning pretrained models
  • · Modular composable architecture: DataProcessor, NegativeMiner and other components can be used independently, with support for custom NegativeMiner integration into training pipelines
View on GitHub

MemAgent

1.1k · Python
Stale B

A MemAgent framework that can extrapolate to 3.5M context tokens, along with a training framework for RL training of any agent workflow.

memoryagentrag +2
  • · Ultra-Long Context Processing: Extrapolate from 8K training context to 3.5M tokens with performance loss under 5%
  • · Reinforcement Learning Driven: Trained with RLVR (Reinforcement Learning from Verifiable Rewards), extends DAPO algorithm for end-to-end multi-turn conversation optimization
  • · Linear Time Complexity: Breaks through computational bottlenecks in long-text processing with linear resource scaling
View on GitHub

Related Articles

RAGChunking检索优化

Advanced RAG: Chunking Strategies and Retrieval Optimization Trade-offs

Most RAG pipelines fail at retrieval, not generation. This article covers five chunking strategies, hybrid search, reranking pipelines, and a production-ready decision framework.

Read more →