Scrapegraph-ai
ActiveDescription
Python scraper based on AI that uses LLMs and knowledge graphs to automatically build web data extraction pipelines.
Key Features
- SmartScraperGraph — Single-page pipeline extracting structured data with just a prompt and URL
- SearchGraph — Multi-page pipeline extracting info from top N search engine results
- ScriptCreatorGraph — Auto-generates Python scraping scripts from webpage content
- Multi-Model Support — Compatible with OpenAI, Ollama, Groq, Azure, Gemini and other LLM backends
- MCP Server Integration — Can serve as an MCP tool callable by AI agents
- Multi-Format Support — Handles XML, HTML, JSON, Markdown and other local document formats
Use Cases
💡 Automatically extracting product prices, reviews, and descriptions from e-commerce websites
💡 Batch scraping search results and structuring market research data
💡 Generating reusable Python scraping scripts for periodic website monitoring
💡 Providing web data extraction tools for AI agents to support RAG knowledge base construction
💡 Processing local HTML/JSON files to extract key information for data analysis
Strengths & Limitations
✅ Strengths
- • Actively maintained, recent updates
- • High community interest (30.4k stars)
- • Permissive open-source license (MIT)
- • Established track record (2 years in production)
Categories
Quick Start
pip install scrapegraphai
playwright install
from scrapegraphai.graphs import SmartScraperGraph
graph_config = {
"llm": {"model": "ollama/llama3.2"},
"verbose": True,
"headless": False,
}
scraper = SmartScraperGraph(
prompt="Extract product info from this page",
source="https://example.com/product",
config=graph_config
)
print(scraper.run())