Prefect: Orchestration Without the DAG
For an Airflow engineer, the surprise isn't a new API — it's that the DAG object is gone and your workflow is just Python that runs.
Prefect for Airflow Engineers
The Pythonic orchestrator, seen from Airflow — flows and tasks, dynamic execution, deployments, work pools, blocks, and Prefect 3's transactions.
-
-
Running Prefect: Your First Flow Run
Install Prefect, write a bookshop flow, and run it by calling the function — then bolt on the UI, profiles, and Cloud when you want them.
-
Flows, Tasks, and Subflows
What earns a @task, why return values replace XComs, and how a flow calling a flow gives you the composition Airflow kept taking away.
-
Dynamic Workflows: Just Write Python
The runtime fan-out that costs you .expand() and a careful read of the docs in Airflow is a for loop and a .map() here.
-
Deployments: From .serve() to Production
A flow you can run is a script. A deployment is the server-side object that turns it into something the platform schedules and triggers for you.
-
Work Pools and Workers
Prefect's scale-out story looks like Airflow's executors turned inside out: the API schedules runs into a pool, and independent workers pull them and provision infrastructure per run — unless the pool is serverless and there's no worker at all.
-
Blocks: Config and Secrets
Where Airflow keeps a connection as untyped key-value pairs and a hook to interpret them, Prefect keeps a typed object that already knows how to be a database.
-
Retries, Timeouts, and the Failures Worth Repeating
Retries are the one Prefect feature that ports straight from Airflow — which is exactly why nobody notices that every retry is a bet on idempotency you never consciously placed.
-
Results and Caching, or: Where Your Return Values Actually Live
If you don't know where your results land, you don't have caching — you have luck. The full cache-policy catalog, the hashing that bites, and the locking nobody configures.
-
Transactions, or: How to Un-Write Half a Load
A flow that half-succeeds leaves your warehouse lying to you, and a retry only makes it worse. Prefect gives tasks commit-or-roll-back semantics — the one thing Airflow has no answer for at all.
-
Automations: The 3 a.m. Runbook That Runs Itself
Ask what Airflow does when a DAG fails and the honest answer is 'nothing you didn't code into the DAG.' Prefect's answer is an event-driven control plane that lives outside your flows.
-
Background Tasks: Your Job Queue Was a Decorator All Along
A Prefect task can run with no flow at all — submitted to the API from a web handler, executed by a task worker somewhere else. This is the feature that walks Prefect straight into Celery's territory.
-
States and State Hooks
The full Prefect state model, Failed versus Crashed, return_state, allow_failure, result handling, and lifecycle hooks.
-
Concurrency and Rate Limiting
Global limits, tag limits, deployment limits, collision strategies, rate_limit, and protecting external systems from dynamic fanout.
-
Variables, Settings, and Profiles
Prefect variables, profile switching, config, PREFECT_API_URL, local versus Cloud, and why blocks are not variables.
-
Pause, Resume, and Human-in-the-Loop Flows
pause_flow_run, suspend_flow_run, wait_for_input, typed input models, and where Prefect has no clean Airflow equivalent.
-
Artifacts and Logging
Prefect artifacts (markdown, table, link, progress) and run-scoped logging with get_run_logger, log_prints, levels, and third-party capture.
-
Testing Prefect Flows
The whole series kept saying it's just a function — testing is where that promise finally pays out, with .fn() for logic and a test harness for orchestration.
-
Cloud or Self-Hosted: Who Gets Paged for the Orchestrator?
This isn't a hosting decision, it's an on-call decision — an unsentimental itemisation of what Prefect Cloud sells you, what self-hosting the server really costs, and the hybrid boundary almost nobody can draw correctly.
-
Migrating from Airflow to Prefect
The finale: a concept-by-concept port of a real Airflow deployment to Prefect 3 — the mapping table, a side-by-side strategy, and the honest gotchas.