Prefect

Prefect is an open-source workflow orchestration framework for building data pipelines in Python. It is designed to be the simplest way to turn a script into a production workflow: you add a few decorators and Prefect takes care of running, tracking and recovering it.

Prefect handles scheduling, caching, retries, dependencies, branching logic and event-based automations, so pipelines can recover from unexpected changes. Workflow activity is tracked and monitored in a self-hosted Prefect server or the hosted Prefect Cloud, with a web UI for runs, logs and deployments.

Prefect is one of the most approachable ways to run data pipelines in production: it stays close to plain Python instead of forcing a new way of writing code. It is Apache-licensed.

Stars

0

Forks

0

Commits

0

Repository

PrefectHQ/prefect

• Apache AirflowWidely used workflow orchestrator for data pipelines.
• AstronomerHosted Apache Airflow service for data teams.
• DagsterNo. 153Open-source data orchestrator built around data assets.
• KestraNo. 072Open-source, event-driven orchestration platform.

See every open-source alternative to Apache Airflow in the collection.