Dagster

Active

GitHub Python Apache-2.0

Description

An orchestration platform for developing, producing, and observing data assets and AI workflows, with built-in asset definitions, scheduling, and monitoring.

Key Features

Declarative asset-based programming model for defining data pipelines as Python functions
Integrated lineage graph with built-in data quality checks and dependency tracking
Web UI for real-time pipeline monitoring, scheduling, and asset health dashboards
Extensive integration library covering cloud warehouses, orchestration tools, and ML frameworks
CI/CD-friendly with local development, unit testing, staging, and production parity
Multi-tenant orchestration engine that scales organizationally with built-in observability

Use Cases

💡 Building and maintaining ETL/ELT data pipelines across cloud data warehouses

💡 Orchestrating ML model training, evaluation, and deployment workflows

💡 Defining and tracking data assets with lineage for regulatory compliance

💡 Coordinating multi-step data processing jobs with scheduling and retries

💡 Managing complex data platform infrastructure with centralized monitoring

Strengths & Limitations

✅ Strengths

• Actively maintained, recent updates
• High community interest (16.1k stars)
• Permissive open-source license (Apache-2.0)

⚠️ Limitations

• High issue backlog (2.6k open issues)

Quick Start

Install via `uv add dagster dagster-webserver dagster-dg-cli`. Define assets as decorated Python functions, load them into the Dagster UI, and use `dagster dev` to launch a local development server with the web interface.

Visit GitHub Visit Website

Related Projects

Prefect

23.8k · Python

Active A

A workflow orchestration framework for building resilient data pipelines and AI workflows in Python, with task scheduling, state management, and failure recovery from local to distributed deployments.

workflowautomationpython +2

· Flow and task decorators — declare Python functions as orchestrable workflows with @flow and @task
· Auto-retry and error handling — built-in task-level retries, caching, parametric execution
· Scheduled deployments — cron expression and event-triggered automated scheduling

View full details → View on GitHub

Instill Core

2.3k · Python

Stale C

A full-stack AI infrastructure tool for data, model, and pipeline orchestration. Streamlines building versatile AI-first applications with a visual pipeline editor for end-to-end workflows from data ingestion to model inference.

workflowpythonautomation +3

· End-to-end AI platform for data, pipeline, and model orchestration
· Visual pipeline editor for building versatile AI-first APIs and automated workflows
· Unstructured data processing turning documents, images, audio, and video into AI-ready formats

View on GitHub

Unstract

7.2k · Python

Active A+

LLM-driven extraction of unstructured data, built for API deployments and ETL pipeline workflows. Automates document parsing, PDF extraction, and intelligent data processing with LLM-powered intelligence.

data-processingragpython +3

· Prompt Studio defines document extraction schemas with natural language
· Supports REST API deployment and ETL pipeline workflows
· MCP Server integration connecting to Claude and other AI agents

View full details → View on GitHub

Integuru

4.8k · Python

Normal A

The first AI agent that builds permissionless integrations through reverse engineering platforms' internal APIs for cross-platform automation.

workflowpythonagent +2

· Automated reverse engineering of platform internal APIs through browser network request analysis
· Dependency graph generation for complex API request chains and authentication flows
· Runnable Python code generation that hits internal endpoints to perform desired actions