Production: The Pipeline That Runs Without You

A green DAG in dev and a pipeline that survives 2am unattended are not the same thing. The difference is idempotency, retries with backoff, alerts that page a human, and a runbook.

Everything up to here runs. That’s not the same as running in production. A DAG that goes green while you watch it is a demo; production is the run that happens at 2am with nobody looking, on the night the warehouse takes an extra thirty seconds to resume, the night ingest fires the trigger twice, the night one source column comes through null. The gap between the two isn’t a bigger machine — it’s four properties a demo lets you skip and an unattended pipeline can’t: it has to be idempotent so a retry or a backfill can’t corrupt the table, it has to retry the failures that are just weather, it has to tell you when it hits one it can’t retry past, and it needs a runbook so the person the pager wakes knows what to do.

This is the orchestration-and-reliability chapter. The other production concerns each got their own: warehouse sizing and resource monitors are chapter 10, CI/CD and artifact promotion are chapter 11, environments and RBAC are chapter 9, and lineage and contracts are chapter 12. This one is about the DAG itself — how you assemble it so that the boring nights stay boring and the bad nights stay recoverable.

The assembled production DAG

You’ve met all the pieces. Chapter 5 wrapped the dbt project in a DbtTaskGroup inside your own @dag. Chapter 6 made the models incremental and swapped the @daily clock for an Asset that ingest updates, so the transform runs because the data arrived, not because a timer went off. Production is where you add the properties that make an unattended run safe, and they all live in the DAG’s arguments:

from datetime import datetime, timedelta

from airflow.sdk import dag, Asset
from airflow.providers.slack.notifications.slack_webhook import SlackWebhookNotifier
from cosmos import DbtTaskGroup, ProjectConfig, ExecutionConfig, RenderConfig
from tpch_profile import profile_config  # the ProfileConfig from chapter 4

raw_orders = Asset("snowflake://analytics_prod/raw/orders")

default_args = {
    "retries": 3,
    "retry_delay": timedelta(minutes=2),
    "retry_exponential_backoff": True,
    "max_retry_delay": timedelta(minutes=20),
    "on_failure_callback": SlackWebhookNotifier(
        slack_webhook_conn_id="slack_alerts",
        text=(
            ":red_circle: `{{ ti.task_id }}` failed in "
            "`{{ dag.dag_id }}` — run `{{ run_id }}`, try {{ ti.try_number }}"
        ),
    ),
}

@dag(
    schedule=[raw_orders],
    start_date=datetime(2026, 6, 1),
    catchup=False,
    max_active_runs=1,
    default_args=default_args,
    tags=["tpch", "prod"],
)
def tpch_prod():
    transform = DbtTaskGroup(
        group_id="dbt_tpch",
        project_config=ProjectConfig("/opt/airflow/dbt/tpch"),
        profile_config=profile_config,
        execution_config=ExecutionConfig(
            dbt_executable_path="/opt/airflow/dbt-venv/bin/dbt"
        ),
        render_config=RenderConfig(select=["staging", "marts"]),
        operator_args={"vars": {"run_date": "{{ data_interval_start | ds }}"}},
    )
    transform

tpch_prod()

Nothing here is new machinery. It’s the chapter-5 DAG with four production dials turned, and every one of them earns its place.

schedule=[raw_orders] is the asset trigger from chapter 6: the ingest DAG lists raw_orders in its outlets, and finishing it wakes this transform. catchup=False says a DAG deployed today with a June start date should not stampede through every day it “missed” — you never want a fresh deploy to launch three weeks of incremental runs on its own.

max_active_runs=1 is the one people forget until it bites. An asset can fire twice in quick succession — ingest reran, or two file batches landed — and without this, Airflow will happily start a second run of the transform while the first is still MERGEing into the marts. Two runs writing the same incremental table on overlapping windows is a race you don’t want to debug. Pinning it to one active run serializes them: the second trigger queues behind the first and starts clean when it finishes.

default_args pushes the retry policy and the failure notifier down into every task Cosmos renders. That’s the quiet advantage of the Cosmos model — because each dbt model and each dbt test is its own Airflow task, retries=3 isn’t “retry the whole dbt build,” it’s “retry this one model three times.” A blip on stg_orders re-runs stg_orders, not the forty tasks that already succeeded.

Check your interval before you trust this. {{ data_interval_start }} is only meaningful if the interval has width — and on a default Airflow 3 install it does not: a cron schedule resolves to CronTriggerTimetable, whose data_interval_start and data_interval_end are the same instant. Any dbt model that windows between them then processes nothing, silently. This track opts in with AIRFLOW__SCHEDULER__CREATE_CRON_DATA_INTERVALS=True; the foundations scheduling chapter explains why and what the alternative is.

The operator_args line passes Airflow’s data_interval_start into dbt as a run_date var. That’s the seam that makes the whole thing backfillable, and it’s worth its own section.

Idempotency across the stack

Idempotency is the property that running a task twice leaves the world in the same state as running it once. It’s what makes a retry safe, and it’s what makes a backfill safe, and in an unattended pipeline you will do both — usually at the same time, usually at night.

Start with the dbt layer, because it’s where the trap hides. Recall the incremental model from chapter 6: it’s materialized incremental with unique_key='order_key', which means dbt doesn’t insert the new window, it MERGEs it — new keys insert, existing keys update in place. That one config is the difference between an idempotent model and a corrupting one. Picture the failure: the task runs, the MERGE commits half a million rows into orders_enriched, and then the worker dies before Airflow marks the task success. Airflow, seeing no success, retries. An append-only model (insert with no unique_key) now inserts those rows a second time — instant duplicates. The MERGE version re-processes the same window, matches the keys it already wrote, and updates them to the identical values. The retry is a no-op you can’t tell happened. That’s idempotency, and you bought it with unique_key.

Backfills lean on the same property from the other direction. In Airflow 3.x, backfills are scheduler-managed — you ask for a date range with airflow backfill create (or the UI) instead of the old CLI daemon, and the scheduler runs those logical dates through the same DAG with the same code. Because you plumbed data_interval_start into a run_date var, each backfilled run tells dbt which slice it is. Key the incremental filter to that var instead of max(order_date):

{% if is_incremental() %}
where o.o_orderdate = '{{ var("run_date") }}'::date
{% endif %}

Now a backfill of June 3rd processes exactly June 3rd’s orders, MERGEs them on order_key, and lands the same rows whether you run it once, run it again, or run it three months after the fact. Contrast the max(order_date) filter from chapter 6: it’s fine for the forward-only daily run, but on a backfill it reads “everything after the latest row already in the table,” which for an old date is nonsense. The data-interval var makes each run reference its own window, and the MERGE makes re-running that window free. This is the through-line: a run should describe the slice it owns, and writing that slice should be repeatable.

catchup=False and idempotency are two ends of the same policy. catchup=False says the pipeline won’t invent runs on its own; idempotency says that when you invent them deliberately — a retry, a backfill, a manual clear-and-rerun — they can’t do damage. You want both. Automatic catchup on an incremental pipeline is a stampede; a deliberate, idempotent backfill is a Tuesday.

Those deliberate reruns come in exactly two flavors, and both are safe only because of the MERGE. The first is the grid’s Clear: click a red task, clear it, and the scheduler re-queues that single task (Airflow can clear downstream tasks with it, so clearing stg_orders re-runs the marts that depend on it too). Because the task rewrites its slice with a MERGE, clearing it a second time — or clearing it after it half-succeeded — converges on the same table. The second is a range: airflow backfill create for, say, June 1–7 runs each of those logical dates through the DAG as its own run, each keyed to its own run_date, each MERGEing its own window. The operator’s daily toolkit is really just those two levers — clear one task, or backfill a range — and the reason you can reach for either at 2am without thinking hard is that you did the thinking once, when you set unique_key and plumbed the interval var. Idempotency isn’t an abstract virtue here; it’s the thing that lets recovery be a click.

The non-dbt tasks have to clear the same bar, or the DAG is only as idempotent as its weakest step. The good news is that the two kinds of bracketing step this stack uses are already idempotent: alter warehouse … set warehouse_size (the resize pattern from chapter 5, whose sizing rationale lives in chapter 10) sets an absolute state, so running it twice is the same as running it once; and Snowflake’s COPY INTO tracks load history per file, so a re-run won’t reload a file it already ingested (the ingestion detail is chapter 8). The step to be suspicious of is any task that increments or appends — a homegrown “log this run” insert, a counter, an email that goes out unconditionally. If a task isn’t naturally idempotent, either make it so (upsert on a run key) or gate it so a retry skips the part that already happened.

Retries: which failures are weather

A retry policy is a bet that most failures are transient. In this stack that bet mostly pays: a warehouse that takes an extra beat to resume, a momentary Snowflake connection reset, a transient lock, a worker the executor reaped. Those are weather. They clear on their own, and a second attempt two minutes later succeeds. The default_args above encode a sane default — three tries, starting at a two-minute delay, doubling each time (retry_exponential_backoff=True) up to a twenty-minute ceiling (max_retry_delay). Exponential backoff matters because the failures that aren’t instant tend to be a system under load, and hammering it every ten seconds makes that worse; backing off gives a resuming warehouse or a throttled account room to recover.

But not every failure is weather, and this is where the Cosmos task graph does you a favor most orchestration setups can’t. Because each dbt model and each dbt test renders as its own task, your grid distinguishes two failures that a monolithic dbt build smears into one exit code:

  • A run task goes red (stg_orders, orders_enriched). The model’s SQL failed to execute, or the warehouse it ran on failed. This is usually infra or a code defect — the kind of thing a retry might paper over (transient) or won’t (a genuine SQL error). Either way it’s a pipeline problem.
  • A test task goes red (stg_orders_order_id_not_null, the relationships test on customer_id). The model built fine; the data it produced violated an assertion. A not_null found nulls; a relationships test found an order pointing at a customer that doesn’t exist. This is a data problem, and it is almost never transient.

That distinction should change how you retry. Retrying a failed test is wasted motion — the test query is deterministic, so re-running it three times just re-discovers the same orphaned key and pages you twenty minutes later than it needed to. If your dbt tests are heavy, it’s worth rendering test tasks with retries=0 while leaving models on the retry policy; Cosmos lets you steer operator arguments by node type, and the payoff is that a data failure alerts fast while an infra flake still gets its three chances. At minimum, know that a red test square means “go look at the data,” not “wait and see.”

One more Cosmos knob belongs here: warn-severity dbt tests. A dbt test set to severity: warn doesn’t fail the task — it passes with a warning — which means by default nobody hears about it. Cosmos surfaces these through an on_warning_callback you can wire to a quieter Slack channel, so the tests you’ve decided aren’t page-worthy still leave a trail instead of vanishing into logs.

Alerting: making the pipeline page you

on_failure_callback fires when a task lands in the failed state, with the run’s context in hand. In the DAG above it’s a SlackWebhookNotifier — Airflow 3.x ships notifiers as first-class, reusable objects, so you attach an instance rather than hand-rolling a callback function. The text field is templated, so every alert names the task, the DAG, the run, and the attempt number — enough for whoever’s on call to open straight to the failed square. PagerDuty is the same shape for the failures that need to wake someone:

from airflow.providers.pagerduty.notifications.pagerduty import PagerdutyNotifier

on_failure_callback = PagerdutyNotifier(
    pagerduty_events_conn_id="pagerduty_prod",
    summary="tpch_prod: {{ ti.task_id }} failed",
    severity="error",
    source="airflow",
)

And when the built-in notifiers don’t fit — you want to route to a team based on which model failed, or enrich the message with the dbt run results — you subclass BaseNotifier and put the logic in notify(context):

from airflow.sdk import BaseNotifier

class RoutedNotifier(BaseNotifier):
    template_fields = ("message",)

    def __init__(self, message):
        self.message = message

    def notify(self, context):
        task_id = context["ti"].task_id
        channel = "#data-oncall" if task_id.endswith("_test") else "#data-infra"
        # dispatch self.message to `channel`
        ...

That task_id.endswith("_test") line is the test-vs-run distinction turned into routing: data failures go to the analysts who own the model, infra failures go to the platform channel. The classification you get for free from the Cosmos graph becomes an on-call policy.

Deadline Alerts replace SLAs. Airflow 3.x removed the old sla= parameter entirely, so the question it used to answer — “what if this finishes late rather than failing?” — needs the new mechanism. A missed-freshness alert is real: the marts feed a 7am executive dashboard, and a run that’s still grinding at 6:55 is a problem even though nothing is red. You attach a DeadlineAlert to the DAG:

from airflow.sdk import DeadlineAlert, DeadlineReference

deadline = DeadlineAlert(
    reference=DeadlineReference.DAGRUN_LOGICAL_DATE,
    interval=timedelta(hours=7),
    callback=SlackWebhookNotifier(
        slack_webhook_conn_id="slack_alerts",
        text=":hourglass: tpch_prod missed its 7am freshness deadline",
    ),
)

Read that as “seven hours after the run’s logical date, if we’re not done, fire the callback.” It’s the SLA idea decoupled from failure: a run can be perfectly healthy and still miss a deadline, and now you hear about the miss on its own channel. Deadline Alerts arrived in the 3.1 line and the exact import path and callback wrapping are still settling across releases — pin them against your own 3.3 build before you rely on them, the way you would any young API.

Finally, alert-storm dedupe, because the first thing a naive setup does on a bad night is send you forty messages. Here the Cosmos graph saves you again, and it’s worth understanding why. When stg_orders fails, the marts downstream of it don’t fail — they go to upstream_failed, a distinct state that means “I never got to run because my dependency broke.” And on_failure_callback fires only on failed, not on upstream_failed. So a failure at the root of a fan-out produces exactly one page (the model that actually broke) plus its own tests, while the ten marts hanging off it stay silent and held back. The storm dedupes itself, as long as your DAG is a real dependency graph and not a pile of independent tasks. For the coarser signal — “the run as a whole failed” — attach a notifier at the DAG level (on_failure_callback on the @dag, not just in default_args) so on-call gets one “pipeline failed” summary alongside the one detailed task page, and nothing in between.

A production runbook

Alerts are worthless if the person they wake doesn’t know the next move. Write the runbook down; here’s the shape of the one this pipeline needs. Each scenario names the chapter that owns the actual fix, because triage lives here and the repair usually lives elsewhere.

A model run task is red. Open the task log — Cosmos shows the compiled SQL and the raw Snowflake error. Read the error first. Warehouse … was suspended or a connection reset is weather; the retries already burned or will, and if the final attempt failed on a resume timeout you clear the single task in the grid and let it rerun (idempotent MERGE, so this is safe). A syntax or column error is not weather — the model’s SQL is broken, which means a change reached production that shouldn’t have, and the fix is upstream in CI, not in the grid (chapter 11). Don’t keep clearing a deterministic failure; it will fail identically every time.

A dbt test task is red. This is a data problem, so resist the reflex to rerun. The model built; its output violated a contract. Inspect the failing rows — if tests are configured to store_failures, the offending records are sitting in a table waiting for you (chapter 12 owns the contract-and-observability tooling). Then decide which of two things happened: the source delivered bad data (an orphaned customer_id, a null where there’s never been one), which is an ingestion conversation (chapter 8); or the world legitimately changed and the test is now wrong, which is a contract update, not a rerun. Either way, notice that the marts downstream are upstream_failed, not published — the test did its job and stopped bad data from reaching the dashboard. That’s a feature. Let it hold.

A source is late. With asset-driven scheduling, “late” looks like nothing — the transform simply never triggered, because the asset it waits on never updated. That’s correct behaviour, not a bug: the pipeline didn’t run on stale data at midnight and quietly publish yesterday’s numbers as today’s. Go look at the ingest DAG (chapter 8) to find out why the source is late. If freshness by a certain hour is contractual, your DeadlineAlert is what should have paged you, not a red square. And do not hand-trigger the transform on stale raw to “get a run in” — you’ll publish incomplete data and MERGE it into the marts, which is exactly the incident the late source was protecting you from.

A key rotated (every task fails at connection). When all tasks fail identically on an authentication error, it’s not the data — it’s the credential. Service-account keys rotate, and if the connection is still pointing at the old private key, nothing authenticates. The fix is a rotation done in one place: register the new public key on the Snowflake user, update the secret in the backend, and the connection resolves the new material on its next run (the secrets backend and role model are chapter 9). Then clear the failed run and let it go — idempotent, so a full rerun is fine.

A run suddenly costs 10x. Nothing is red; the DAG is green and the bill isn’t. Orchestration didn’t break — a model did, quietly. The usual culprit is an incremental filter that stopped filtering (a bad merge, a changed source column) so a “delta” run scanned the entire history. The Airflow grid tells you which task got slow; Snowflake’s query history and query tags tell you why and what it cost, and resource monitors are what should have caught it before the invoice did — all chapter 10. The orchestration lesson that stays here: a green run is not a healthy run, and duration is a signal you should be watching, not just success.

Final thoughts

Look back at what actually turned the chapter-5 demo into a production pipeline, and none of it was a new tool. It was four arguments and a document. max_active_runs=1 and catchup=False keep the DAG from getting in its own way. unique_key on the incremental models and a data-interval var make every run describe and rewrite its own slice, so a retry or a backfill can’t corrupt the table. A retry policy with backoff absorbs the failures that are just weather, while the Cosmos task graph splits the ones that aren’t into “the pipeline broke” and “the data broke” — a distinction that flows straight into who gets paged. And the runbook turns each of those pages into a decision instead of a panic.

That’s the real definition of production for a data stack: not that it never fails, but that its failures are legible. A red test square means look at the data; a red run task means look at the pipeline; a run that never fired means look upstream; a green run that costs a fortune means look at the query history. Every failure mode has a shape, a signal, and a chapter that owns the fix. Build the pipeline so those shapes stay distinct, and 2am stops being the hour you dread — it’s just another idempotent run, retrying its way through the weather, paging you only when it truly can’t cope.

Next: The Ingestion Layer

Comments