Your First DAG: Tasks That Know Their Order

Write a two-task bookshop pipeline in Python and watch the scheduler run it in order.

A DAG is a Python file. Not a config format that happens to be Python-flavored — an actual module the scheduler imports, where the objects you create become the graph. Drop a file in dags/, and within seconds Airflow parses it, finds the DAG inside, and starts scheduling it. Let’s write the bookshop’s daily load, see the ordering enforced for real, and then take apart every knob on the DAG that decides when and how often that ordering plays out.

The file

Create dags/bookshop.py:

from datetime import datetime
from airflow.sdk import dag, task


@dag(
    schedule="@daily",
    start_date=datetime(2026, 5, 1),
    catchup=False,
)
def bookshop():
    @task
    def extract_orders() -> list[dict]:
        return [
            {"title": "Dune", "qty": 2, "price": 18},
            {"title": "Piranesi", "qty": 1, "price": 15},
        ]

    @task
    def summarize_orders(orders: list[dict]) -> None:
        revenue = sum(o["qty"] * o["price"] for o in orders)
        print(f"{len(orders)} orders, revenue {revenue}")

    summarize_orders(extract_orders())


bookshop()

That’s a complete, runnable DAG. Three things are doing the work.

@dag(...) turns the bookshop function into a DAG definition. schedule="@daily" runs it once a day; start_date is when scheduling begins; catchup=False says don’t replay every day since May 1st the moment you turn it on. This is the 3.x TaskFlow API — you write plain Python functions and decorate them, and the imports come from airflow.sdk, the Task SDK that’s the authoring home in 3.x. (If you’ve read older tutorials, the imports were from airflow import DAG and from airflow.decorators import dag. In 3.x it’s one namespace: airflow.sdk.)

Each @task function is a task. And the last line inside bookshop is where the ordering lives: summarize_orders(extract_orders()). You’re not calling these functions now — the decorator intercepts the call and records that summarize_orders depends on the output of extract_orders. Passing one’s return into the other is how you draw the edge. Extract runs first, its return value flows into summarize, and Airflow guarantees that order without a sleep or a lock file in sight.

Save the file. Because dags/ is mounted into the containers, the scheduler re-parses within a few seconds and bookshop shows up in the UI on its own. But before you trigger it, the three-line decorator hides a fleet of choices worth making on purpose.

Every knob on @dag

The three arguments above are the minimum. Here’s the same DAG with the parameters you’ll actually reach for in production, and a plain-English gloss on each:

from datetime import datetime, timedelta
from airflow.sdk import dag, task

default_args = {
    "owner": "data-eng",
    "retries": 2,
    "retry_delay": timedelta(minutes=5),
}


@dag(
    dag_id="bookshop_daily",
    description="Daily extract-and-summarize of bookshop orders",
    tags=["bookshop", "daily", "finance"],
    doc_md=__doc__,
    default_args=default_args,
    schedule="@daily",
    start_date=datetime(2026, 5, 1),
    end_date=None,
    catchup=False,
    max_active_runs=1,
    dagrun_timeout=timedelta(hours=1),
)
def bookshop_daily():
    ...


bookshop_daily()

Reading top to bottom:

  • dag_id — the unique name across your whole Airflow instance. When you don’t pass it, the decorator uses the function name (bookshop in the first version). Set it explicitly the moment you care what the URL and the logs call it; two DAGs with the same id is an error, and the id is what the CLI and REST API address.
  • description — one line shown next to the DAG in the list view. Keep it to what the pipeline does, not how.
  • tags — free-form labels. They become clickable filters in the UI, which matters more than it sounds: once you have forty DAGs, tags=["bookshop"] is how you find yours in one click.
  • doc_md — Markdown rendered on the DAG’s page in the UI. Passing __doc__ wires it to the module’s docstring, so a triple-quoted block at the top of the file becomes on-screen documentation. This is where the why goes.
  • default_args — a dict of defaults handed to every task in the DAG. More on this next; it’s the single most useful parameter here.
  • schedule — how often a run fires. "@daily", a cron string like "0 6 * * *", a timedelta, None for manual-only, or a list of Assets for event-driven runs. In 3.x this one argument replaces the old schedule_interval and timetable — there is now just schedule=.
  • start_date — the first date the scheduler will consider. Its interaction with schedule is subtle enough to get its own section below.
  • end_date — an optional date after which no new runs are created. None means “run forever,” which is what you want almost always.
  • catchup — whether to backfill every missed interval between start_date and now when the DAG is first turned on. Leave it False unless you specifically want history replayed; the scheduling post covers the case where you want it True.
  • max_active_runs — how many DAG runs may execute concurrently. 1 forces runs to go strictly one at a time, which is the safe default for a pipeline that writes to the same tables each day — you don’t want yesterday’s slow run and today’s run both loading into orders at once.
  • dagrun_timeout — a wall-clock ceiling on a single run. If the whole DAG hasn’t finished within this timedelta, the run is marked failed and its running tasks are killed. Without it, a wedged task can hold a run open indefinitely and, with max_active_runs=1, block every run behind it.

There’s a matching non-decorator form using a context manager — with DAG(dag_id=..., schedule=...) as dag: — that takes the exact same arguments. You’ll see it in the operators post and in a lot of existing code. The decorator and the context manager are two spellings of the same object; pick the decorator for TaskFlow-heavy DAGs and don’t lose sleep over the choice.

default_args is where retries live

Notice that owner, retries, and retry_delay went into default_args, not onto @dag directly. That placement is deliberate, and it trips up newcomers who go looking for a retries= argument on the DAG itself.

The distinction: @dag parameters configure the DAG — its schedule, its concurrency, its timeout. default_args configures the tasks, by supplying a default that every task in the DAG inherits unless it overrides. Retries are a per-task concern (a task fails and retries; a DAG doesn’t), so retries belongs in default_args.

The usual residents of default_args:

  • owner — who’s responsible for the task. Shows up in the UI and in logs; useful when a shared Airflow runs several teams’ DAGs.
  • retries — how many times to re-run a task after it fails before giving up. 2 means up to three attempts total.
  • retry_delay — how long to wait between attempts, as a timedelta. Five minutes is a sane default for anything that talks to a flaky external service.
  • callbacks — on_failure_callback, on_success_callback, and on_retry_callback are functions Airflow calls when a task changes state. This is where SLA-style alerting lives in 3.x: the old dedicated SLA feature was removed, and the replacement is a callback plus your own timing check. Wire on_failure_callback to a function that posts to Slack and every task in the DAG gains failure alerting for free.

Because default_args is inherited, you set the team’s policy once at the top and every task obeys it. A single task can still override — @task(retries=5) on one flaky extractor — and its value wins over the DAG default for that task alone. Think of default_args as the DAG’s house rules and the per-task argument as a signed exception.

start_date: the parameter people get wrong

start_date looks obvious and isn’t. Three things reliably bite.

Naive vs timezone-aware datetimes. datetime(2026, 5, 1) is naive — it carries no timezone. Airflow interprets a naive start_date in the DAG’s configured timezone (UTC by default), which is fine as long as you know that’s happening. If your business runs on a specific zone, make it explicit and unambiguous with a timezone-aware datetime:

import pendulum

start_date = pendulum.datetime(2026, 5, 1, tz="Europe/London")

Airflow ships pendulum precisely for this. An aware start_date removes the guesswork about which midnight “May 1st” means, and it’s what the UI and logs will echo back to you.

Never compute start_date dynamically. The single most common start_date bug:

# WRONG — do not do this
start_date=datetime.now() - timedelta(days=1)

Remember that the DAG file is re-parsed constantly — every few seconds. datetime.now() evaluates fresh on every parse, so your start_date keeps sliding forward in real time. The scheduler compares “now” against a moving target and can never decide a run is due, so the DAG mysteriously never fires. start_date must be a fixed, static point in the past — a literal date you typed, not a computed one. Pin it and forget it.

How start_date and schedule combine to produce the first run. Airflow schedules based on data intervals. With start_date=datetime(2026, 5, 1) and schedule="@daily", the first interval covers all of May 1st — from 00:00 May 1 to 00:00 May 2. And here’s the part that surprises everyone: Airflow runs an interval after it has completed, not at its start. So the run for the May 1st interval fires just after midnight on May 2nd. The DAG “for the 1st” executes on the 2nd, because only then is the 1st’s data complete and ready to process.

(That behavior is the data-interval model, which this series turns on with one line of config — the scheduling chapter explains why, and what Airflow 3 does by default instead. It matters more than it sounds.) This is the right behavior once it clicks — you process a day’s data after the day is over — but it means a DAG with start_date set to today won’t produce its first run until tomorrow. If you toggle a brand-new DAG on and nothing happens, this is usually why, and it’s not a bug. (The scheduling post goes deep on data intervals; for now, just hold the shape: interval ends, then the run for it fires.)

Three words that aren’t the same thing

The UI uses three terms constantly and conflates none of them:

A DAG is the definition — the file you just wrote, the template. It doesn’t “run”; it describes what a run looks like.

A DAG run is one execution of that template for a particular data interval — the @daily run for May 24th. Trigger it three times and you have three DAG runs of one DAG.

A task instance is one task within one DAG run — extract_orders for the May 24th run. This is the thing that actually has logs, a duration, and a state.

When you stare at a grid of colored squares later, each square is one task instance in one state. Nothing more mysterious than that — but there are more states than the grid’s colors first suggest.

The full task-state lifecycle

A task instance moves through a state machine, and knowing the whole map is what turns a red square from a mystery into a diagnosis. Picture the flow from left to right:

     none → scheduled → queued → running → success
                            │        │  \
                            │        │   └──→ deferred ──┐
                            │        │                   │ (event/timer)
                            │        │  ┌────────────────┘
                            │        │  └→ up_for_reschedule ──┐
                            │        ▼                         │
                            │      failed → up_for_retry ──────┤
                            │        │                         │
   upstream_failed ◄────────┘        └→ (retries exhausted) failed
   skipped
   removed / restarting

Walking the states you’ll actually see:

  • none — the task exists in the parsed DAG but the scheduler hasn’t looked at it yet for this run. The starting point.
  • scheduled — the scheduler has decided this task’s dependencies are met and it should run. It’s waiting for an executor slot.
  • queued — handed to the executor, sitting in the queue, waiting for a worker to pick it up. On a busy instance a task can sit here a while; a long queued time means you’re out of worker capacity, not that the task is slow.
  • running — a worker is actively executing it. This is the only state where your code is on a CPU.
  • success — finished cleanly (exit 0, no exception). Green.
  • failed — raised an exception or exited non-zero, with no retries left. Red.
  • up_for_retry — failed, but has retries remaining. The scheduler holds it for retry_delay, then sends it back to scheduled for another attempt. This is the state that makes a transient network blip self-heal.
  • deferred — the task handed itself off to the triggerer and is waiting on an external event or timer without occupying a worker slot. This is how modern sensors wait efficiently; it gets its own post later. The key idea now: deferred is “parked, costing nothing,” not “stuck.”
  • up_for_reschedule — a sensor running in reschedule mode has checked, found its condition not yet met, and released its slot to try again later. Like deferred in spirit, older in mechanism.
  • skipped — deliberately not run, usually because a branch chose a different path. Not a failure; an intentional no-op.
  • upstream_failed — a task this one depends on failed (or was skipped), so this task can’t run and won’t be attempted. The failure propagates downstream: one red task upstream paints a row of upstream_failed behind it. When you’re debugging, always look left for the first genuinely failed task — everything to its right is just collateral.
  • removed — the task was in an earlier version of the DAG but no longer exists in the current parse, while a run that included it is still around. You edited the DAG and deleted a task; its history lingers as removed.
  • restarting — the task was asked to clear and run again while it was still running; it’s being torn down to be relaunched.

Two habits fall out of this map. First, up_for_retry and deferred are not problems — a DAG full of them is working as designed. Second, when a run goes red, the fix always starts at the leftmost failed, never at the upstream_failed squares trailing it.

Drawing the edges: every dependency form

summarize_orders(extract_orders()) draws an edge by passing data. But you’ll often want to order tasks that don’t pass data — “run the cleanup after the load, regardless.” Airflow gives you several spellings, and they all build the same graph.

The bitshift operators are the everyday form. >> means “then”:

extract >> transform >> load

Read left to right: extract, then transform, then load. << is the same arrow reversed (load << transform << extract), which reads backwards and which I’d avoid for exactly that reason.

Lists fan out and fan in. A list on either side of >> means “all of these”:

extract >> [validate, archive]      # extract, then both, in parallel
[validate, archive] >> notify       # after both finish, notify

The first line makes validate and archive both depend on extract; the second makes notify wait for both. Chain the two and you’ve built a diamond.

For longer sequences, chain reads more cleanly than a wall of >>, and it handles parallel stages elegantly:

from airflow.sdk import chain

chain(extract, [transform_a, transform_b], load)

That means: extract, then transform_a and transform_b in parallel, then load after both. chain also has a rule worth knowing — when it connects two equal-length lists, it pairs them element-wise, not cross-wise: chain([a, b], [c, d]) wires a >> c and b >> d, and nothing else.

When you do want the cross product — every task on the left connected to every task on the right — that’s cross_downstream:

from airflow.sdk import cross_downstream

cross_downstream([extract_orders, extract_returns], [load_orders, load_returns])

This makes all four edges: each extract feeds each load. It’s a common shape when several inputs must all land before several outputs can start.

Finally, the method form. Every task has set_upstream and set_downstream:

transform.set_upstream(extract)     # same as extract >> transform
transform.set_downstream(load)      # same as transform >> load

These are what >> and << call under the hood. You’ll rarely reach for them by hand — the operators are terser — but they’re occasionally handy when you’re wiring dependencies inside a loop and the objects are already in variables. Whichever spelling you use, the result is one thing: edges in the graph the scheduler walks.

What “parsing” actually is

Everything above hinges on a word I’ve used loosely: parse. Here’s what it means concretely, because it explains the one line that most first-timers forget.

A separate Airflow component — the dag-processor, distinct from the scheduler in 3.x — watches the dags/ folder and, on a loop, imports each Python file as a module. Importing runs the file top to bottom, like python bookshop.py would. As it runs, your @dag-decorated function and the bookshop() call at the bottom construct DAG objects in memory, and the processor serializes those into the metadata database. The scheduler then reads DAGs from the database — it never imports your file itself. That serialized copy is what the UI renders and the scheduler schedules.

Which is exactly why the bare bookshop() at the bottom of the file is mandatory. @dag turns your function into a factory that produces a DAG when called — but if you never call it, no DAG object is ever built, nothing gets serialized, and your file parses without error into precisely zero DAGs. Forget that line and your DAG silently never appears, no traceback to point you at the cause. It’s the single most common first-time stumble. (The context-manager form sidesteps this — with DAG(...) as dag: constructs the object immediately — which is one reason some people prefer it.)

Because the file is executed on every parse, and parses happen every few seconds, top-level code is a trap. Anything at module scope — outside a task function — runs on every single parse, on the dag-processor, not on a worker. A line like this at the top of the file:

orders = duckdb.sql("SELECT * FROM huge_table").df()   # DON'T

runs constantly, in the parsing process, whether or not the DAG is even scheduled. It’ll hammer your database on a schedule you didn’t choose — the dag-processor re-parses each changed file every [dag_processor] min_file_process_interval seconds (default 30) and rescans the folder every refresh_interval seconds (default 300) — slow every parse (which delays every DAG showing up), and it can’t produce anything a task can use anyway. The rule: top-level code should only define the graph, never do the work. All the real work — the queries, the API calls, the file reads — goes inside @task functions, which run on workers only when the task actually executes. If you find yourself doing I/O at module scope, you’ve put the work in the wrong place.

One more discovery detail: not every .py in dags/ should be parsed. Helper modules, work-in-progress files, vendored code — you exclude them with a .airflowignore file in dags/. It works like .gitignore: one pattern per line, and any file path matching a pattern is skipped by the dag-processor entirely.

# dags/.airflowignore
_wip_*
utils/

Mind the syntax, because it changed and the internet hasn’t caught up. In Airflow 3 .airflowignore defaults to glob patterns ([core] dag_ignore_file_syntax = glob); in 2.x the default was regex. They look deceptively similar and they are not: _wip_.* is the regex for “starts with _wip_”, and as a glob it means “starts with _wip_.” — a literal dot. Paste the regex form into a 3.x project and _wip_orders.py is cheerfully parsed anyway, which is the opposite of what you asked for and produces no complaint at all. Write _wip_*, or set the syntax back to regexp if you have a pile of inherited patterns.

That keeps half-written DAGs and shared helpers from being imported, failing, and cluttering the UI with import errors.

Run it and read it

Back to the working DAG. In the UI, find bookshop, toggle it on (new DAGs start paused), and hit the trigger button for a manual run. The scheduler queues extract_orders, a worker runs it, its return value is stashed for the next task, then summarize_orders runs. Two squares go from grey to green.

The two views you’ll live in:

  • Grid — runs as columns, tasks as rows, every cell a task instance colored by state. This is your at-a-glance history: which runs passed, which task in which run went red.
  • Graph — the DAG’s shape, extract_orders → summarize_orders, showing the dependency you declared. This is where you confirm the wiring is what you meant.

Click summarize_orders in either view and open its logs. You’ll see your 2 orders, revenue 51 printed there — task output goes to that task instance’s log, not your terminal, which is exactly where you’ll go first when a task goes red in the posts ahead.

Final thoughts

The move that makes Airflow click is realizing the dependency isn’t configuration you write about your code — it’s the shape of the code itself. summarize_orders(extract_orders()) reads like an ordinary function call, and that’s the point: the graph and the Python are the same text. But the file is doing double duty, and holding both halves in your head is what separates a DAG that works from one that mystifies you. Top to bottom it’s a program the dag-processor executes on every parse — which is why start_date must be static, why the bare bookshop() call is load-bearing, and why heavy work at module scope is a mistake. Inside the task functions it’s a description of work that runs later, on a worker, in the order the edges dictate. Everything from here — operators, XComs, scheduling — is elaboration on those two ideas living in one file.

Next: Operators: The Verbs of a DAG

Comments