HomeProjectsArticlesPK BoardAbout
Explore Projects

Crawlee

Active
GitHub TypeScript Apache-2.0

Description

A web scraping and browser automation library for Node.js to build reliable crawlers, supporting Puppeteer, Playwright, Cheerio, and raw HTTP. Extract data for AI, LLMs, RAG, or GPTs with proxy rotation and both headful and headless modes.

Key Features

  • Multi-engine unified interface — same API supports Puppeteer, Playwright, Cheerio and raw HTTP
  • Anti-detection stealth — default config simulates human behavior to bypass major bot protections
  • Proxy rotation and session management — built-in proxy rotation, fingerprint generation and session persistence
  • Persistent queue — supports breadth-first and depth-first URL crawling queues
  • Automatic scaling — adjusts crawling concurrency based on available system resources
  • Structured data storage — built-in Dataset and KeyValueStore for tabular and file storage

Use Cases

💡 Batch scrape web content for AI/RAG applications to build knowledge base data sources
💡 Crawl e-commerce product info for price monitoring and competitive analysis
💡 Automate news and social media content collection for sentiment analysis
💡 Extract structured data from JavaScript-rendered SPA websites
💡 Build scheduled crawling pipelines to monitor data changes on target websites

Strengths & Limitations

✅ Strengths

  • • Actively maintained, recent updates
  • • High community interest (25.6k stars)
  • • Permissive open-source license (Apache-2.0)
  • • Responsive to issues, low backlog

Tags

typescript javascript data-processing tools rag automation

Categories

📚 RAG Tools ⚡ Agent Tools

Quick Start

npx crawlee create my-crawler
cd my-crawler
npm start
Visit GitHub

Related Projects

Parsr

6.2k · JavaScript
Stale B

Transforms PDF, documents and images into enriched structured data with table recognition, reading order restoration, and Markdown output.

javascriptragtools +2
  • · Multi-format document parsing: supports PDF, DOCX, images, and EML files with minimal footprint
  • · Structured output in JSON, Markdown, CSV/Pandas DataFrame, and TXT formats
  • · Document structure detection: headings, tables, lists, table of contents, page numbers, headers/footers, and links
View full details → View on GitHub

Docstrange

1.5k · Python
Stale B

Extract and convert data from any document (PDFs, images, Word, PPT, URLs) into multiple formats including Markdown, JSON, and CSV.

pythonragtools +2
  • · Multi-format conversion — transforms PDF, DOCX, PPTX, XLSX, images, and URLs into Markdown, JSON, CSV, and HTML
  • · 7B parameter model — upgraded core model for significantly higher accuracy and deeper document understanding
  • · Advanced OCR pipeline — extracts text from scanned documents, phone photos, and receipts with high accuracy
View on GitHub

WrenAI

17.5k · Python
Active A

Open-source text-to-SQL and text-to-chart GenBI agent with a semantic layer. Ask your database questions in natural language and get accurate SQL, charts, and BI insights. Supports 12+ data sources and any LLM.

llmtypescriptagent +2
  • · Open context layer providing business semantics, examples, and governance for AI agents
  • · Agent-driven design with CLI-embedded workflow guides loaded on demand
  • · Modeling Definition Language (MDL) for models, relationships, cubes, metrics, and RLAC/CLAC
View full details → View on GitHub