Dagster

Dagster is a cloud-native data pipeline orchestrator for the whole development lifecycle. You declare the data assets you want, such as tables, datasets, machine learning models and reports, as Python functions, and Dagster runs them at the right time and keeps them up to date. Prefect is acquiring Dagster Labs, the company behind it; Dagster keeps its name and open-source license.

Dagster has a declarative programming model, integrated lineage and observability, and is built for testability. Assets and their dependencies show as a graph in its web UI.

Dagster brings software engineering practices like testing and lineage to data pipelines. It is Apache-licensed, with a hosted Dagster+ option.

Stars

0

Forks

0

Commits

0

Repository

dagster-io/dagster

• Apache AirflowWidely used workflow orchestrator for data pipelines.
• AstronomerManaged Apache Airflow platform.
• KestraNo. 072Open-source, declarative orchestration platform.

See every open-source alternative to Apache Airflow in the collection.