Schedules, Catchup, and Backfills Without the Fear

A daily DAG that runs at midnight is processing yesterday, not today. Once that clicks, schedules, catchup, backfills, timezones, and Assets stop being scary.

You’ve written a DAG. Now you have to tell Airflow when it should run — and this is the point where a lot of people quietly develop a superstition about the scheduler. They deploy something, twelve runs appear out of nowhere, and they conclude the tool is haunted. It isn’t. It’s doing exactly what you asked, and once you understand the time model it will never surprise you again.

This is the longest chapter in the foundations series, and on purpose: scheduling is where the mental model earns its keep. Get it right and every later topic — retries, backfills, data-aware pipelines — is just this idea applied again.

The many ways to say “when”

The schedule= argument takes a few shapes, and in Airflow 3.x this is the only place scheduling lives. If a tutorial reaches for schedule_interval= or a separate timetable= keyword, it’s written for 2.x — both were removed and folded into this one unified argument.

from datetime import timedelta

schedule="@daily"                 # a preset
schedule="0 6 * * *"              # a cron string — 06:00 every day
schedule=timedelta(hours=4)       # a fixed cadence
schedule=None                     # never on a clock; you trigger it

The presets are aliases for cron strings, and the list is longer than most people remember:

  • @hourly → 0 * * * *
  • @daily → 0 0 * * * (midnight — worth saying out loud, because people assume “daily” means “some sensible business hour”)
  • @weekly → 0 0 * * 0 (Sunday midnight)
  • @monthly → 0 0 1 * *
  • @quarterly → 0 0 1 */3 *
  • @yearly (a.k.a. @annually) → 0 0 1 1 *
  • @once — run exactly one time, ever, then never again
  • @continuous — start a new run the instant the previous one finishes, with no clock at all; useful for a near-real-time poll loop. It also requires max_active_runs=1 and refuses to load otherwise (ValueError: Invalid max_active_runs: ContinuousTimetable requires max_active_runs <= 1) — which is the scheduler saving you from the obvious foot-gun, since the whole point is that runs never overlap

A cron string gives you the precision cron always did. A timedelta says “one interval is this long”; None means the DAG has no clock — you run it by hand, or another DAG’s output triggers it (that’s Assets, which we get to properly at the end of this chapter, not with a hand-wave). And @continuous is the odd one out: no interval boundaries, just back-to-back runs.

For the bookshop’s daily order load, @daily is right. But daily hides the question that trips everyone up, so let’s drag it into the light.

Two models of “when,” and the one Airflow 3 picked

This is the section that will save you a genuinely nasty afternoon, so read it slowly. Airflow has two ideas of what a scheduled run means, Airflow 3 changed which one is the default, and almost everything written about Airflow before 3.0 assumes the other.

The trigger model — Airflow 3’s default. A run fires at its scheduled instant. A @daily DAG fires at midnight, and that run’s logical_date is midnight of that day. Simple, and exactly what a newcomer expects a scheduler to do.

The data-interval model — Airflow 2’s default, still available. A run represents a window that has closed. A @daily DAG’s run for June 5th covers all of June 5th, and therefore cannot start until June 5th is over — so it fires just after midnight on June 6th, stamped with a logical date of June 5th.

That second model is the source of the most famous confusion in Airflow’s history (“why is my midnight run processing yesterday?”), and it’s why Airflow 3 flipped the default. But here is the part nobody tells you, and it is the whole reason this section exists: under the trigger model, the data interval is zero-width. Not “yesterday.” Not “one day.” Zero:

# Airflow 3 defaults: [scheduler] create_cron_data_intervals = False
# schedule="@daily"  ->  CronTriggerTimetable
#
#   run fires 2026-06-05 00:00   data_interval_start = 2026-06-05 00:00
#                                data_interval_end   = 2026-06-05 00:00   <-- same instant

Now look at what that does to the most natural query a batch pipeline ever writes:

WHERE order_date >= '{{ data_interval_start | ds }}'
  AND order_date <  '{{ data_interval_end | ds }}'
-- renders to:  >= '2026-06-05' AND < '2026-06-05'

Start equals end, the half-open window is empty, and the query matches zero rows. No error. No warning. A green DAG that loads nothing, every night, until someone notices the dashboard has been flat for a month. If you take one thing from this chapter, take this: on Airflow 3, data_interval_start and data_interval_end are the same value for a cron schedule unless you ask for otherwise.

What this book does, and why

This series is a windowed batch book. Its whole running example is “load yesterday’s orders” — a fixed window, processed once, backfillable. For that shape of work the data-interval model isn’t legacy baggage, it’s the correct abstraction: each run owns one window, the windows tile time with no gaps and no overlaps, and a backfill of last June replays June’s windows rather than thirty copies of today. So we turn it on, explicitly, in one line:

# .env
AIRFLOW__SCHEDULER__CREATE_CRON_DATA_INTERVALS=True

With that set, @daily gets a CronDataIntervalTimetable, the run for June 5th fires just after midnight on June 6th, and the two context values describe the window it owns:

from airflow.sdk import task

@task
def load_orders(data_interval_start, data_interval_end):
    # for the June 5th run, with data intervals enabled:
    #   data_interval_start = 2026-06-05 00:00+00:00
    #   data_interval_end   = 2026-06-06 00:00+00:00
    day = data_interval_start.strftime("%Y-%m-%d")
    ...

Every templated window in the rest of this series assumes that setting. If you’d rather stay on the 3.x default, you can get the same window arithmetic from logical_date — {{ macros.ds_add(ds, -1) }} to {{ ds }} is “the day that just finished” — but then you own the off-by-one, on every DAG, forever. Pick one, write it down, and don’t mix them.

The test that settles it in five seconds, on any DAG you inherit: print both bounds. If they’re equal, you’re on the trigger model, and any query that windows between them is silently returning nothing.

Reach for data_interval_start / data_interval_end whenever a task needs to know which day it’s processing. Never reach for datetime.now() — we’ll see why in the backfill section.

The date macros you’ll actually reach for

Those two interval values are also available as Jinja templates, which is how you feed them into SQL, file paths, and shell commands. The full chapter on templating comes later; here’s the scheduling-relevant subset, because a schedule you can’t reference from a query isn’t much use.

For the June 5th bookshop run, these render as:

TemplateValueWhat it is
{{ ds }}2026-06-05logical date, YYYY-MM-DD
{{ ds_nodash }}20260605same, no dashes — great for partition folders
{{ ts }}2026-06-05T00:00:00+00:00logical date as a full ISO timestamp
{{ data_interval_start }}2026-06-05T00:00:00+00:00window opens
{{ data_interval_end }}2026-06-06T00:00:00+00:00window closes
{{ prev_data_interval_start_success }}prior run’s startthe last successful run’s interval start
{{ macros.ds_add(ds, -7) }}2026-05-29date arithmetic in a template

{{ ds }} is the workhorse — it’s the day being processed, as a string, ready to interpolate:

from airflow.providers.standard.operators.bash import BashOperator

pull = BashOperator(
    task_id="pull_orders",
    bash_command="load_orders --for-date {{ ds }} "
                 "--out s3://bookshop/orders/{{ ds_nodash }}/orders.parquet",
)

prev_data_interval_start_success is the sneaky-useful one. It skips failed runs, so an incremental load can say “give me everything since the last time I actually succeeded” without re-processing a window twice or leaving a hole because the previous night errored out. And macros.ds_add(ds, n) (with macros.ds_format alongside it) lets you do the “seven days back” arithmetic inside the template instead of smuggling a Python helper into bash_command.

One more thing that surprises people: a manual run has a data interval too. When you hit “Trigger” in the UI on a scheduled DAG, Airflow asks the timetable to compute an interval for that trigger — so ds and the interval macros are still populated, not blank. On a DAG with schedule=None, though, there’s no interval to compute, so the window collapses to a single instant: data_interval_start == data_interval_end, both equal to the trigger time. A task written to assume a full day between start and end will quietly produce an empty window on those DAGs. If your logic depends on the interval having width, don’t run it with schedule=None.

Timezones, or: the naive-datetime pitfall

Everything above showed intervals in UTC, and that’s not cosmetic — Airflow computes and stores all scheduling times in UTC. Your logical_date, your data intervals, the timestamps in the metadata database: UTC, always. The timezone only matters at two edges — the start_date you write, and the wall-clock moment a cron fires.

The trap is writing a naive datetime:

from datetime import datetime

start_date = datetime(2026, 5, 25)   # naive — no timezone attached

Airflow will assume UTC and warn you, and for a UTC shop that’s harmless. But the moment your schedule needs to mean a local time — “midnight in New York,” “6 a.m. in London” — a naive datetime is a latent bug. Use pendulum, which Airflow ships with and uses internally, and attach a real zone:

import pendulum
from airflow.sdk import dag

@dag(
    schedule="0 0 * * *",              # local midnight
    start_date=pendulum.datetime(2026, 5, 25, tz="America/New_York"),
    catchup=False,
)
def bookshop_daily():
    ...

Now “midnight” tracks New York’s wall clock, and here’s the part that earns pendulum its place: DST is handled for you. A cron-based schedule anchors to the local wall clock, so 0 0 * * * fires at 00:00 New York time whether the offset is currently -05:00 or -04:00. Airflow does the conversion to UTC per run — which means the UTC logical dates of your runs shift by an hour across the spring and fall transitions, even though the local time you asked for never moved. That’s correct, and it’s exactly what you want for a report that has to land “by 9 a.m. local” year-round.

The one schedule that does not respect wall-clock time is timedelta. A timedelta(hours=24) cadence counts 24 real hours of elapsed time, so across a DST boundary it lands an hour off the local clock and stays there. If you care about local time-of-day, use cron; if you care about a fixed duration between runs, use timedelta — and know which one you’re asking for.

Custom timetables: when the presets aren’t enough

Under the hood, every schedule you’ve seen is a timetable — an object that answers two questions: given the last run, what’s the next data interval, and when does it become eligible to run. The string and timedelta forms are shorthands Airflow expands into built-in timetable objects. When you need behavior the shorthands can’t express, you pass a Timetable instance to the same schedule= argument:

from airflow.timetables.trigger import CronTriggerTimetable
from airflow.timetables.interval import DeltaDataIntervalTimetable
import pendulum

# Fire at the cron instant, stamped with that instant.
schedule=CronTriggerTimetable("0 9 * * 1-5", timezone="America/New_York")

# A rolling four-hour window, anchored to start_date.
schedule=DeltaDataIntervalTimetable(timedelta(hours=4))

The two are worth contrasting, because they encode two different intentions:

  • CronTriggerTimetable fires at the cron time and stamps the run with that time. There’s no “processing the previous window” story — a 0 9 * * 1-5 trigger at 9 a.m. Monday is a run for 9 a.m. Monday. Naming it explicitly is mostly a way of being loud about your intent, because this is already what a bare cron string gives you on a default Airflow 3 install (that’s the trigger model from the top of this chapter). Reach for it when the run is a task at a moment, not a batch over a window — and remember its data interval is zero-width.

  • DeltaDataIntervalTimetable gives a timedelta cadence with a real window: intervals are contiguous, each new one starting exactly where the last ended, delta wide, anchored to start_date. Note that it is not what a bare schedule=timedelta(hours=4) produces — that resolves to DeltaTriggerTimetable, the zero-width sibling, unless you set create_delta_data_intervals. Naming DeltaDataIntervalTimetable is how you ask for the window.

That last one hides the nuance the docs bury. It’s tempting to read timedelta(hours=4) as “run four hours after the previous run finishes.” It doesn’t measure from completion. The next interval opens where the previous interval ended, not where the previous run’s tasks happened to wrap up. So if a run takes three hours, the next window still opens on the four-hour grid, not seven hours after you started. If the scheduler was down and misses a beat, the intervals resume from the last interval boundary, not from “now.” The cadence is anchored to the interval math, never to how long your job ran — which is the whole reason a delayed run doesn’t smear your entire schedule.

Custom timetables get deeper than this — you can write one that skips holidays, or aligns to a fiscal calendar — but that’s the advanced series. For foundations, the point is: schedule= accepts an object, and the two built-ins above cover most of what the presets can’t.

Catchup: the twelve mysterious runs

Now the haunting. You write the bookshop DAG on June 6th, but you set start_date to May 25th because that’s when the data begins. You deploy. Twelve runs appear and start grinding through history.

That’s catchup, and it defaults to False in modern Airflow for a reason. With catchup=True, the scheduler looks at every interval between start_date and now, notices none of them have run, and fills them all in — one run per missed day.

Let’s make it concrete. Deploy this at noon on June 6th:

@dag(
    schedule="@daily",
    start_date=pendulum.datetime(2026, 5, 25, tz="UTC"),
    catchup=True,
    max_active_runs=2,
)
def bookshop_daily():
    ...

The scheduler enumerates the closed daily intervals from May 25th through June 5th — twelve of them — and queues a run for each. It won’t run all twelve at once: max_active_runs=2 caps it at two concurrent runs, so they drain two-at-a-time in interval order until history is caught up, then the DAG continues normally. Without that cap, twelve runs would all hit your warehouse simultaneously, which is how a catchup takes down a database at lunch.

Two settings shape how that catch-up marches:

  • max_active_runs throttles concurrency — how many of those historical runs execute at the same time. Always set it when catchup is on.
  • depends_on_past=True (a task setting, next section) forces order — a task instance won’t start until the same task in the previous run succeeded. Combined with catchup it turns the fill-in into a strict oldest-to-newest march, which is what you want if June 5th’s numbers depend on June 4th’s already being loaded.

There’s also LatestOnlyOperator, the escape hatch for tasks that should not be part of a catchup. Put it upstream of, say, a “send the Slack summary” task, and during a backfill or catchup those downstream tasks are skipped for every run except the most recent interval. You backfill the data quietly without firing eleven historical Slack alerts at your team.

So when is catchup=True genuinely right? When missing an interval is a real gap you must fill automatically, and every run is cheap and idempotent — a metrics rollup that must exist for every day, where a hole in the series is a bug. Even then, most teams keep catchup=False and fill history deliberately — which is exactly what backfills are for.

depends_on_past and wait_for_downstream

Two task-level flags decide whether runs are allowed to overtake each other in time. They’re easy to confuse and they solve different problems.

depends_on_past=True ties a task to itself in the previous run. Task instance for June 5th won’t start until the June 4th instance of the same task succeeded. Use it when a run genuinely builds on the last one — an incremental model that reads yesterday’s output, a running balance, anything where processing days out of order corrupts the result. The catch: one failure stalls everything after it. If June 4th errors and you don’t fix it, June 5th onward never start. That’s the feature working, but it means you can’t ignore a red run.

wait_for_downstream=True is the stronger, rarer cousin. It says a task in this run waits until all downstream tasks of the same task in the previous run have completed — not just the task itself. You’d want this when the current step would clobber data that the previous run’s later steps still need to read. It implies depends_on_past, and it serializes your DAG hard, so reach for it only when you can name the specific corruption it prevents.

Default both to off. Turn depends_on_past on when order is correctness, not preference; turn wait_for_downstream on almost never.

Backfills are the scheduler’s job now

If you’ve read older Airflow material, “backfill” means a command you ran in a terminal — airflow dags backfill — that spun up a long-lived process on your laptop and executed the range itself, outside the scheduler, for as long as your SSH session survived. That’s gone.

In 3.x a backfill is a request to the scheduler. You name a DAG and a date range, and the scheduler creates and runs those runs the same way it runs scheduled ones — same executor, same queue, same visibility in the UI. Nothing depends on your terminal. You start it from the UI (there’s a “Backfill” action on the DAG page with a date-range form) or from the CLI:

airflow backfill create \
  --dag-id bookshop_daily \
  --from-date 2026-05-25 \
  --to-date 2026-05-31 \
  --max-active-runs 3

The flags are where the control lives:

  • --max-active-runs throttles the backfill’s own concurrency, independent of the DAG’s normal setting. This is your safety valve against thirty runs hammering the warehouse — set it low for a heavy pipeline.
  • --run-backwards processes the range newest-to-oldest instead of oldest-to-first. Handy when the recent days matter most and you want them refreshed first, provided your tasks don’t have depends_on_past (which requires forward order and will conflict).
  • --reprocess re-runs intervals that already have a run, instead of skipping them. Without it, a backfill only fills in missing runs; with it, you force existing runs to execute again — the flag you want after fixing a bug that made last week’s numbers wrong.

The runs show up in the grid, throttle against your limits, retry on failure, and you watch them from the browser instead of a scrollback buffer. Backfilling a quarter of history used to be an act of nerve; now it’s a form and a progress view.

Designing a DAG you can safely backfill

A backfill re-runs old intervals, which means it re-executes tasks that already ran. Everything good about backfills depends on your tasks not caring — and that comes down to two rules.

First, be idempotent: running the June 5th load twice must leave the warehouse identical to running it once. Overwrite the day’s partition, or delete-then-insert for that interval — never blind-append, or a backfill will double your rows.

Second, and this is the one that bites, parametrize by the data interval, never by datetime.now(). A task that computes “today” from the wall clock is fine on its first live run and catastrophic on backfill: ask it to reprocess May 25th and it cheerfully loads today’s file into May 25th’s slot. Read the day from data_interval_start or {{ ds }}. Then the same code produces the same result whether it runs live tonight or backfills a year from now — which is the entire point of an orchestrator that knows what time it’s pretending to be.

Data-aware scheduling: Assets

Every schedule so far has been a clock. But the bookshop has a second kind of “when”: run the sales report as soon as the orders are loaded — not at a fixed time, but when a specific dataset is fresh. That’s a data-aware schedule, and in Airflow 3.x it’s built on Assets. (If you’ve used Airflow 2.4–2.10, these were called Datasets; same idea, renamed.)

An Asset is a named handle for a piece of data — usually a URI, though the string is just a stable identifier, not something Airflow reads. A DAG produces an asset by listing it as an outlet; another DAG consumes it by using it as its schedule. Here’s the producer half of the bookshop:

from airflow.sdk import Asset, dag, task
import pendulum

orders_asset = Asset("s3://bookshop/orders/daily.parquet")

@dag(schedule="@daily", start_date=pendulum.datetime(2026, 5, 25, tz="UTC"), catchup=False)
def bookshop_ingest():
    @task(outlets=[orders_asset])
    def load_orders(data_interval_start):
        # write today's orders to S3...
        return

    load_orders()

bookshop_ingest()

When load_orders finishes successfully, Airflow records that orders_asset was updated. That update is an event the scheduler watches. Now the consumer:

from airflow.sdk import dag, task

@dag(schedule=[orders_asset], catchup=False)
def bookshop_report():
    @task
    def build_sales_summary():
        # read the fresh orders and roll them up...
        return

    build_sales_summary()

bookshop_report()

Note what’s not there: no cron, no start_date doing scheduling work, no clock at all. schedule=[orders_asset] says “run whenever this asset updates.” The instant bookshop_ingest marks the asset fresh, the scheduler triggers bookshop_report. If the report depended on several inputs — orders and the day’s returns — you’d list them all: schedule=[orders_asset, returns_asset], and by default the DAG runs once all of them have updated. No more guessing at a cron offset generous enough to clear the upstream job; the dependency is explicit, and if ingestion runs late, the report simply waits.

There’s also a decorator shorthand when a DAG is really just “produce one asset”:

from airflow.sdk import asset

@asset(schedule="@daily")
def orders():
    # this function's body produces the asset named `orders`
    return

@asset defines a single-task DAG whose output is the asset — the function name becomes the asset, and downstream DAGs can schedule on it exactly as above. It’s the concise form for the common “one job, one dataset” case.

That’s the foundations-level view: assets connect DAGs by data instead of time, so a pipeline can be event-driven end to end. Cross-DAG lineage, conditional asset expressions (A | B for “either”), asset aliases, and watchers for external systems are all real and all deferred to the advanced series — but the promise this series kept dangling is now delivered: you can schedule a DAG on another DAG’s output, today, with the two snippets above.

Final thoughts

Scheduling in Airflow feels mystical for exactly as long as you think a run is “the job running now.” Once you internalize that a run owns an interval and fires after that interval closes, the rest collapses into common sense: catchup is “fill the intervals I missed,” backfill is “fill the intervals I choose,” timezones are a UTC core with local edges, timetables are just objects that compute intervals, and Assets swap the clock for an event. All of it is safe precisely when your tasks read time from the interval instead of the wall clock. Get that one habit right and you’ll never again write a task that quietly loads today’s data into last month’s table.

Next: Connections and Hooks: Talking to the Outside World

Comments