Prefect: Orchestration Without the DAG

In Airflow, the graph comes first. You declare a DAG, the scheduler imports the file and parses the objects into a static shape, and only then does anything run. When you want the shape to change with the data — a task per file, a branch that depends on a row count — you reach for dynamic task mapping, branching operators, and Jinja templating, because the graph was fixed at parse time and you’re negotiating with it after the fact.

Prefect turns that inside out. There is no DAG object to declare and no parse step to satisfy. You write a Python function, decorate it @flow, and call it — and the running of that function is the workflow. The graph isn’t a thing you build up front for a scheduler to inspect; it’s whatever your code happens to do this time, discovered as it executes. If you’ve spent a while bending Airflow’s static graph to fit dynamic work, that one difference is most of the story.

What Prefect actually is

Prefect is a Python-native orchestrator. Two decorators carry the core:

from prefect import flow, task

@task
def extract_orders() -> list[dict]:
    return [
        {"title": "Dune", "qty": 2, "price": 18},
        {"title": "Piranesi", "qty": 1, "price": 15},
    ]

@flow(log_prints=True)
def daily_load():
    orders = extract_orders()
    revenue = sum(o["qty"] * o["price"] for o in orders)
    print(f"{len(orders)} orders, revenue {revenue}")

if __name__ == "__main__":
    daily_load()

Run that file and it runs. No dags/ folder to be scanned, no scheduler process that has to be alive, no toggling the flow on in a UI first. A flow is an ordinary Python function that Prefect has wrapped with orchestration — state tracking, logging, retries — and a task is a unit inside it that gets the same treatment. Flows call tasks, and flows can call other flows. That last one matters more than it looks: composition is just a function calling a function, which is exactly the thing Airflow made awkward.

The same pipeline, both ways

Description only gets you so far. Here is a bookshop load — extract orders, compute revenue — written first the way you’d write it in Airflow, then in Prefect, so the swap is visible line for line.

The Airflow version, against 3.x:

from datetime import datetime
from airflow import DAG
from airflow.providers.standard.operators.python import PythonOperator

def extract_orders(**context):
    orders = [
        {"title": "Dune", "qty": 2, "price": 18},
        {"title": "Piranesi", "qty": 1, "price": 15},
    ]
    context["ti"].xcom_push(key="orders", value=orders)

def compute_revenue(**context):
    orders = context["ti"].xcom_pull(key="orders", task_ids="extract")
    revenue = sum(o["qty"] * o["price"] for o in orders)
    print(f"{len(orders)} orders, revenue {revenue}")

with DAG(
    dag_id="daily_load",
    start_date=datetime(2026, 1, 1),
    schedule="@daily",
    catchup=False,
) as dag:
    extract = PythonOperator(task_id="extract", python_callable=extract_orders)
    revenue = PythonOperator(task_id="revenue", python_callable=compute_revenue)
    extract >> revenue

The Prefect equivalent:

from prefect import flow, task

@task
def extract_orders() -> list[dict]:
    return [
        {"title": "Dune", "qty": 2, "price": 18},
        {"title": "Piranesi", "qty": 1, "price": 15},
    ]

@task
def compute_revenue(orders: list[dict]) -> float:
    return sum(o["qty"] * o["price"] for o in orders)

@flow(log_prints=True)
def daily_load():
    orders = extract_orders()
    revenue = compute_revenue(orders)
    print(f"{len(orders)} orders, revenue {revenue}")

if __name__ == "__main__":
    daily_load()

Read the two side by side and every Airflow concept has quietly dissolved into plain Python. The with DAG(...) block is gone — the @flow function is the DAG. The two PythonOperator wrappers are gone — the functions are the tasks. The extract >> revenue edge is gone — the dependency exists because compute_revenue(orders) takes the output of extract_orders() as an argument, so Prefect knows one must finish before the other starts. And the whole xcom_push/xcom_pull dance — pushing a value into a side-channel keyed by task id, pulling it back out by name — collapses into orders = extract_orders(). The data moves the way data moves in Python: it’s a return value you hold in a variable.

The schedule hasn’t vanished, it’s just not baked into the workflow file. In Airflow the schedule="@daily" lives on the DAG object because the DAG and its schedule are the same declaration. In Prefect the flow doesn’t know or care how often it runs; scheduling is a property of a deployment, set separately, which is why the same flow can be a scheduled batch job, an ad-hoc python daily_load.py, and an event-triggered run without touching its code. We’ll get to deployments in their own post — for now the point is that the scheduling concern has been peeled off the workflow definition.

The noun-for-noun map

Most of learning Prefect from Airflow is re-mapping a vocabulary you already own. Here is the whole table in one place; the rest of the series is really just filling in each row.

Airflow	Prefect	What changed
DAG	@flow function	The function is the workflow; no separate graph object to build.
Operator / task	@task function	You write Python and decorate it instead of choosing an operator class.
>> / set_upstream	Passing a return value	Order comes from data flow between function calls, not declared edges.
XCom	Task return value	A plain Python return, held in a variable — no push/pull, no key.
Connection	Block	A typed, validated config object (pydantic v2), stored and reusable.
Variable	Variable	Same idea, prefect.variables.Variable; JSON values by key.
Executor (Celery / K8s)	Work pool + task runner	Where runs execute (work pool) split from how tasks fan out in-process (task runner).
Sensor	Event + Automation	Instead of a task that polls, an external event triggers a run via an automation.
Pool / concurrency slot	Work pool / work queue / concurrency limit	Throughput is capped at the pool, the queue, or with named concurrency limits.
SubDAG / TaskGroup	Subflow	A flow calling another flow — ordinary composition, not a special construct.
Scheduler	Prefect server + deployment schedule	The schedule lives on a deployment, not on the workflow.
Webserver	Prefect UI (server or Cloud)	Records what happened; the flow can run without it being alive.
Backfill / catchup	Deployment run (prefect deployment run)	Re-runs are triggered runs with parameters, not a catchup window.

The Connection → Block and Variable → Variable rows are the ones you’ll touch first when you move a real pipeline over, so a taste of both. An Airflow connection is a row in a metadata table addressed by a conn_id string, unpacked at runtime by a hook. A Prefect block is a typed, validated object you define, register, and load by name:

from prefect import flow
from prefect.variables import Variable
from prefect_sqlalchemy import SqlAlchemyConnector  # a stock block type

@flow
def load_to_warehouse():
    warehouse = SqlAlchemyConnector.load("bookshop-dw")   # was a Connection
    batch_size = Variable.get("load_batch_size", default=500)  # was a Variable
    with warehouse.get_connection(begin=False) as conn:
        ...

The difference from a conn_id string is that the block is a pydantic v2 model — its fields are typed and validated when you save it, and misspelling a field fails loudly at registration rather than with a KeyError three tasks deep. Variable.get(...) is the near-identical twin of airflow.models.Variable.get, JSON values by key, with a real default argument instead of a sentinel. Blocks get a whole post later; the mapping is clean enough that this is most of it.

Two rows deserve a flag because they trip people up. Executor → work pool + task runner is not one-to-one: Airflow’s executor bundles two concerns Prefect keeps apart. The work pool decides where a flow run lands — a local process, a Docker container, a Kubernetes job, a serverless push pool — and a worker pulls runs from it. The task runner is a separate, in-flow choice about whether the tasks inside a single run execute sequentially, in threads, or on a Dask/Ray cluster. You can run tasks concurrently on your laptop with no infrastructure, or run each flow on Kubernetes with tasks sequential inside it; those are independent dials. And Sensor → event + automation is a genuine inversion: an Airflow sensor is a task that sits in the graph pulling — occupying a slot, poking a condition on a schedule. Prefect flips it to push — something emits an event, an automation matches it, and a run is triggered. Nothing sits and waits burning a worker.

Prefect 2 was a different animal

If you go looking for tutorials, most of the search results are Prefect 2, and enough changed in 3.x that following them will strand you. The differences worth knowing before you read anything dated 2023 or 2024:

• Agents are gone. Prefect 2 used agents — lightweight pollers tied to a queue — to pick up scheduled runs. Prefect 3 replaced them entirely with typed work pools and workers. If a guide tells you to prefect agent start, it’s out of date; the command and the concept no longer exist.
• Deployments are built differently. The 2.x Deployment.build_from_flow(...) API and the prefect deployment build CLI are gone. In 3.x you use flow.serve() for the simple case, flow.deploy() for containerized/remote work, or a prefect.yaml with prefect deploy for repo-driven deployments.
• Blocks are pydantic v2. Prefect moved to pydantic v2 across the board, and blocks — the typed config objects — are pydantic v2 models. Custom blocks written against v1 validators need updating.
• Results and transactions are new. Prefect 3 added first-class, durably persisted results and transactions — with transaction(): blocks with on_rollback hooks, so a group of tasks commits or rolls back together. There is no Airflow equivalent and no Prefect 2 equivalent; it’s the headline 3.x feature, and chapter 10 is built around it.
• Autonomous background tasks are new. Calling some_task.delay(...) to run work on a separate task server, outside any flow, is a 3.x capability.

There’s a subtler shift underneath results, too: in 3.x a task’s return value can be persisted to configurable storage and keyed, which is what makes caching (“don’t recompute this”) and transactions (“commit or roll back these together”) possible at all. Prefect 2 treated results as mostly in-memory plumbing; Prefect 3 treats them as durable, addressable artifacts. If a 2.x tutorial’s mental model of caching feels thinner than what the UI shows you, that’s why.

The through-line is that Prefect 3 is a real re-architecture, not a point release, with a rewritten engine and a stronger concurrency and transactional story. Treat 2.x material as archaeology.

The moving parts

Because Prefect drops the always-on scheduler, it’s easy to assume there’s no server at all. There is — you just don’t always need it running. Four pieces make up a Prefect deployment:

• The API server — a FastAPI service (prefect server start) that is the orchestration brain: it stores flow-run and task-run state, holds schedules, and serves the REST API every other piece talks to.
• The database — where that state lives. SQLite by default for local development; Postgres for anything real. This is your run history, your logs, your block and variable storage.
• The UI — a React app the server hosts, showing flow runs, task runs, states, logs, and (once a run exists) its graph. It’s an observability surface, not the thing that makes runs happen.
• Workers — processes that poll a work pool for scheduled runs and execute them. Pull-based: a worker asks the API “anything for me?” rather than a scheduler shoving work at it.

That last piece is the quiet architectural swap. Airflow’s scheduler pushes: a central process holds the whole graph, decides what’s runnable, and hands work to executors. It’s the brain and the bottleneck — if it falls behind, everything waits, and scaling it is its own tuning exercise. Prefect’s workers pull: the API just records that a run is due, and independent workers ask for work when they have capacity. Nothing central has to know how many workers exist or where they run, so you can point a Kubernetes work pool and a laptop process pool at the same API and each pulls only what it’s meant to. The failure modes are gentler, too — a worker dying doesn’t stall the API, and standing up more throughput is starting another worker, not retuning a scheduler.

The part that feels alien coming from Airflow is that a bare python daily_load.py still works with none of this stood up. In that mode Prefect spins up a temporary ephemeral API backed by a throwaway local SQLite, tracks the run, and tears it down when the script exits — so you get states, retries, and logging with zero infrastructure. Point the same code at a persistent server (prefect server start) or at Prefect Cloud and now the history sticks around, the UI has something to show, and deployments can schedule the flow. Ephemeral mode is how you develop; a persistent API is how you operate. The workflow code is identical either way.

That persistent API comes in two editions, which the deployments and work-pool posts will lean on:

• Self-hosted (open source) — free, prefect server start, you run the API, a Postgres, and your workers. Single-tenant, and the auth story is thin but not absent: it ships with a single shared credential — set PREFECT_SERVER_API_AUTH_STRING="admin:hunter2" and both the UI and the API demand it (unauthenticated calls get a 401). What it does not have is per-user identity, RBAC, or SSO — it’s one password for everyone, so you’re still expected to put it behind your own network controls. “No auth” is the thing people say; “one shared password and no roles” is the truth, and the difference matters when you’re deciding how much to trust it.
• Prefect Cloud — a hosted API and UI with workspaces, users, RBAC, SSO, service accounts, audit logs, event-driven automations, and serverless push work pools that run flows without you managing a worker at all. There’s a free tier; paid plans scale the seats, retention, and governance.

The practical rule of thumb: reach for self-hosted when you’re learning, prototyping, or running inside a single trusted network where you can live without built-in auth — it’s free and it’s honest about being single-tenant. Reach for Cloud the moment more than one person needs scoped access, you want event-driven automations and audit trails without building them, or you’d rather not babysit a Postgres and a fleet of workers. Crucially, your flow code doesn’t change between the two — you’re only pointing it at a different API URL — so this isn’t a lock-in decision you have to make on day one. Start ephemeral, graduate to a self-hosted server when you want history to persist, and move to Cloud when governance or serverless execution earns its cost.

Where it wins, and where Airflow still leads

This is the series’ framing, so let’s be honest rather than sell. Prefect wins on the dynamic, Pythonic end. Workflows whose shape depends on runtime data are a plain for loop and an if, not a mapping API. There’s far less ceremony between an idea and a running pipeline, and testing a flow is testing a function — you can call it in a pytest file with no test harness for a scheduler. If your work is genuinely data-shaped and changes run to run, Prefect stops fighting you.

Airflow still leads where its maturity compounds, and it’s worth naming the specific gaps rather than waving at them:

• The provider ecosystem is much smaller. Airflow has hundreds of operators and hooks; Prefect has a tidy set of integration libraries (prefect-aws, prefect-gcp, prefect-snowflake, prefect-dbt, and friends) and expects you to write more plain Python against vendors’ own SDKs. That’s often fine — a task calling boto3 is not hard — but the “there’s already an operator for this” reflex won’t always pay off.
• RBAC is thin outside Cloud. The open-source server has no real access control; meaningful RBAC, SSO, and audit logs are Cloud (paid) features. Airflow’s OSS RBAC is more mature.
• Cloud pricing is a real line item. The managed edition is usage-based, and at scale — many flows, high run volume, long retention — it can surprise you. Budget for it if Cloud is the plan.
• The graph is only visible at runtime. This is the flip side of “no static DAG.” Because the shape is discovered as the code runs, there’s no parse-time diagram of a pipeline you haven’t executed yet. The UI draws the graph while and after it runs, which is great for observability and worse for eyeballing an unfamiliar flow’s structure cold. Airflow’s static graph, for all its rigidity, is a picture you can read before anything happens.
• The ecosystem around it is younger. Fewer managed offerings (there’s no first-party AWS/GCP managed Prefect the way MWAA and Cloud Composer wrap Airflow — Prefect Cloud is the managed option), fewer Stack Overflow answers, less institutional muscle memory on teams.

If you haven’t yet, the case for any orchestrator is worth reading in the Airflow foundations post on why orchestration exists — Prefect answers the same need, it just answers it in Python instead of in a graph you declare.

And Prefect isn’t the only one making this bet. If you’re evaluating, know the neighbors. Dagster is asset-first: you declare the data assets you want to exist and it orchestrates to produce them, which maps cleanly onto a dbt-shaped world and gives you a lineage catalog out of the box — more opinionated than Prefect, and the closest thing to “orchestration that thinks in tables.” Temporal is a durable-execution engine for long-running, mission-critical workflows — sagas, microservice coordination — polyglot and lower-level, heavier to operate, and not really data-pipeline-shaped (though Prefect’s transactions borrow its durable-execution ideas). Mage is a batteries-included, UI-forward pipeline tool aimed at getting data engineers productive fast, less flexible than code-first Prefect but quicker to a first result. Prefect sits where you want general, code-first orchestration in Python with dynamic workflows and minimal ceremony — and where a static asset graph or a durable microservice engine would be either too opinionated or too low-level.

What the series covers

Sixteen posts, built for someone who already thinks in DAGs, grouped by theme rather than read strictly in order:

• The model and the setup — this post’s swap, running Prefect and standing up the server UI, and flows/tasks/subflows as the building blocks of composition.
• Making runs happen — dynamic workflows where fan-out is just a loop; deployments from .serve() to something production schedules and triggers; and work pools and workers, where and how runs execute at scale.
• Config and control — blocks for typed config and secrets, variables/settings/profiles, states and state-change hooks, and concurrency and rate limiting.
• Durability and failure — results and caching so you don’t recompute what you already have, and retries, timeouts, and the exact semantics of failure.
• The Prefect 3 frontier — pause/resume and human-in-the-loop flows, and transactions, automations, and autonomous tasks — the features with no Airflow equivalent.

By the end you should be able to read a Prefect codebase and see the Airflow shape underneath it, and know which of your pipelines are better off written this way.

Final thoughts

The DAG object was never the point of orchestration — retries, observability, scheduling, and not running step two before step one are. Airflow reached those through a graph you declare and a scheduler that parses it, and that machinery earned it a decade of ubiquity. Prefect makes a bet that for a large class of Python-heavy, dynamic pipelines, the declared graph was accidental complexity — that if the language already expresses order through ordinary function calls, the orchestrator should read the order off the running code instead of asking you to write it twice. Whether that bet pays off for your pipelines is a real question, and it’s the one the rest of this series is built to help you answer.

Next: Running Prefect: Your First Flow Run