Only Rebuild What Changed

Once the bookshop project grows, rebuilding every model on every run is waste — dbt's state comparison lets you build only what a change actually touched, if you get the manifest plumbing right.

Cosmos gave you a task per model, selection gave you the ability to run a slice on purpose. Both still assume you know, ahead of time, which slice matters. But the most valuable selection isn’t one you write — it’s one dbt works out for you by noticing what changed. The bookshop has six models today; give it a year and it has sixty, and rebuilding all sixty at 2am because someone edited one stg_ model is the kind of waste that’s invisible until the warehouse bill arrives.

This chapter is about two dbt features that lean on the same artifact — --state and --defer — and about the unglamorous plumbing that makes them work under Cosmos. A warning up front on vocabulary, because it will bite you otherwise: Airflow has its own thing called a deferrable operator, which hands a wait off to the triggerer process to free a worker slot. That is not what “defer” means here. In this chapter “deferral” always means the dbt --defer flag, which is about resolving ref()s against production. Same five letters, completely unrelated machinery. I’ll flag the collision again at the end so it sticks.

What “changed” means to dbt

dbt can compare your current project against a snapshot of a previous run and tell you exactly which nodes differ. That snapshot is the manifest.json — the artifact dbt writes to target/ on every invocation, describing every model, its compiled SQL, its config, and its place in the graph. Point dbt at a directory containing a prior manifest and the state: selector method comes alive:

dbt build --select "state:modified+" --state ./prod-artifacts

state:modified resolves to every node whose definition differs from the manifest in ./prod-artifacts — new models, edited SQL, changed configs, altered tests. The trailing + is the graph operator you already know: also build everything downstream, because a change to int_order_payments isn’t done until orders and customers have seen it. Edit one staging model and dbt selects that model and its descendants, and nothing else. The untouched half of the project doesn’t run.

This is the same idea as --select from the last post, with the selection computed from a diff instead of typed by hand. Everything downstream of the machinery is unchanged: dbt still runs a graph, Cosmos still renders that graph as Airflow tasks.

--state compares; --defer resolves

These two flags travel together so often that people fuse them into one concept, and then get confused when one works without the other. They do different jobs.

--state is comparison. It’s the flag that gives state:modified a baseline to diff against. It answers “what is different?” and nothing more. You can use it entirely on its own — a nightly run that selects state:modified+ and rebuilds that subtree in a schema that already has every upstream table needs --state and doesn’t need --defer at all, because every ref() in the selected subtree resolves to a table that’s really there.

--defer is resolution. It changes what a ref() points at at compile time. Normally ref('int_order_payments') compiles to “the int_order_payments relation in the schema I’m building into.” With --defer, if int_order_payments isn’t part of this run’s selection, dbt rewrites that reference to point at the version described in the state manifest — i.e. production — instead of expecting a freshly built copy in your scratch schema.

Here’s where you need both, and why. State selection has a gap. Say you edit only orders, a marts model. state:modified+ correctly picks orders and nothing upstream. But orders does ref('int_order_payments'), and in a fresh CI schema that intermediate table was never built, so the compile fails resolving a reference to something you deliberately didn’t select. --defer closes the gap:

dbt build --select "state:modified+" --defer --state ./prod-artifacts

Now orders reads from the real, already-built int_order_payments in the production schema, and only orders itself gets rebuilt. This is the mechanism that makes slim CI honest — a pull request can build the two models it touched, deferring every unchanged ref() to prod, without cloning the entire warehouse into a scratch schema first. Note that --defer uses the same --state manifest to know where production’s objects live, which is why the two feel welded together: --state supplies the baseline, and both state:modified (comparison) and --defer (resolution) read from it.

--favor-state, and the case where defer surprises you

There’s a subtlety in how --defer decides whether to use production. By default, deferral only kicks in when the referenced relation doesn’t exist in your target schema. If your CI schema happens to still hold a stale int_order_payments from a previous PR run, a plain --defer will happily use that stale copy instead of prod — because the relation exists, so dbt sees no reason to reach for the deferred one. That’s a real source of “it passed in CI, it’s wrong in prod” mysteries.

--favor-state flips the preference. With it, dbt uses the deferred (production) relation for any node that isn’t in the current selection even if a copy exists in your target schema:

dbt build --select "state:modified+" --defer --favor-state --state ./prod-artifacts

Reach for --favor-state whenever your scratch schema is long-lived enough to accumulate leftovers — shared dev schemas, reused CI namespaces. It says “trust prod over whatever junk is lying around in my schema,” which is almost always what you want the moment a schema outlives a single run.

When state:modified cries wolf

state:modified is a coarse instrument, and its favorite failure mode is over-reporting. The comparison is over the compiled node, so anything that changes compiled SQL marks a model modified — including changes that don’t change a single row of output.

The classic trap is a shared macro. Say the bookshop has a cents_to_dollars macro that every model touching money calls — stg_payments, int_order_payments, orders, customers. You rename its argument from column_name to col for readability. Not a single output value changes. But dbt recompiles all four models with different SQL text, so state:modified flags all four — plus, via +, everything downstream of them, which for the bookshop is most of the marts layer. Reformat your whole project with a SQL formatter and every model is “modified.” Bump a dbt_utils version that alters generated SQL, and the blast radius is the entire project. State selection was supposed to shrink the run; a whitespace-only PR can make it rebuild the world, and it’ll do it with total confidence because, at the level dbt compares, the SQL genuinely did change.

dbt gives you finer sub-selectors so you can ask what kind of change happened rather than treating all diffs alike:

  • state:modified.body — the model’s SQL body changed (this is the one you usually care about).
  • state:modified.configs — a config changed (materialization, tags, cluster_by, etc.).
  • state:modified.contract — the model’s enforced contract changed (columns, types).
  • state:modified.relation — the database relation it points to changed (name, schema, database).

There are more (.persisted_descriptions, .macros, and so on), but those four carry most of the weight. Selecting state:modified.body+ instead of the blanket state:modified+ narrows the surface to models whose compiled body actually differs — a PR that only edited a description: in a YAML file, or retagged some models, stops triggering a rebuild of their downstream trees. It doesn’t save you from the cents_to_dollars rename, though: macros expand into the body, so a macro edit still shows up as a .body change on every caller. That’s not a bug in the selector — the body really is different — it’s the selector being honest about a change you happen to know is cosmetic. In practice the honest answer to macro churn isn’t a cleverer selector; it’s discipline about when you reformat (do it in its own PR, expect a full rebuild, move on) and, for genuinely cosmetic changes, being willing to just run the full build that once rather than trusting a diff you know is noisy. State comparison is a heuristic, not an oracle. It’s usually right and occasionally spectacularly wrong, and knowing which is on you.

Making the manifest reach Cosmos

Everything above hinges on one artifact being in the right place at the right time. State selection is only as good as the manifest you compare against — so the actual work is managing that file, not writing the selector.

The pattern: after each successful production run, publish that run’s target/manifest.json somewhere durable — object storage is the usual home — under a stable key. A subsequent run pulls that manifest down to a fixed local path, hands it to dbt as the --state directory, and selects state:modified+ against it. The comparison is always “current project vs. last known-good production.”

Under Cosmos there’s a wrinkle that trips people badly, and it comes from when Cosmos reads the manifest. Selection flows through RenderConfig, which with LoadMode.DBT_LS shells out to dbt ls to decide which nodes become tasks. That dbt ls runs at DAG parse time, on the scheduler — not at task-run time on a worker. So the state manifest has to be present, on the scheduler’s filesystem, before the scheduler parses the DAG file. A pre-task that downloads it inside the DAG run is too late: by the time that task fires, rendering already happened (or failed) minutes or hours earlier during parsing.

Concretely, stage the manifest with a hook — here’s the S3 version — writing it to a path both the scheduler (at parse) and the workers (at run) can read:

# include/stage_manifest.py
from airflow.providers.amazon.aws.hooks.s3 import S3Hook

STATE_DIR = "/usr/local/airflow/include/dbt/state"
STATE_PATH = f"{STATE_DIR}/manifest.json"

def stage_prod_manifest():
    """Pull the last known-good production manifest to a fixed local path."""
    import os
    os.makedirs(STATE_DIR, exist_ok=True)
    hook = S3Hook(aws_conn_id="aws_default")
    key = "dbt/bookshop/prod/manifest.json"  # published by the prod run on success
    obj = hook.get_key(key, bucket_name="bookshop-artifacts")
    obj.download_file(STATE_PATH)
    return STATE_PATH

The GCS equivalent is the same shape with GCSHook(...).download(bucket_name=..., object_name=..., filename=STATE_PATH). Now the awkward part: this needs to run on the scheduler before it parses the DAG, and DAG files are parsed on a loop, not on a schedule you control. The clean way to satisfy that is to make manifest-staging a step that runs outside and ahead of the DAG that renders against it — most teams do it as a small dedicated DAG (or a deploy-time step) that refreshes include/dbt/state/manifest.json on the scheduler on its own cadence, and the slim DAG simply reads whatever’s there when it next parses:

from cosmos import DbtDag, ProjectConfig, RenderConfig, ExecutionConfig
from cosmos.constants import LoadMode

STATE_DIR = "/usr/local/airflow/include/dbt/state"

render = RenderConfig(
    load_method=LoadMode.DBT_LS,
    select=["state:modified.body+"],
    # dbt ls runs at PARSE TIME on the scheduler; the manifest below
    # must already be at STATE_DIR when the scheduler parses this file.
    env_vars={"DBT_STATE": STATE_DIR},
)

DbtDag(
    project_config=ProjectConfig("/usr/local/airflow/include/dbt/bookshop"),
    render_config=render,
    execution_config=ExecutionConfig(dbt_project_path="/usr/local/airflow/include/dbt/bookshop"),
    profile_config=profile_config,
    operator_args={
        # --state alone only tells dbt what to COMPARE against. Deferral is a
        # separate switch, and Cosmos has no `defer` parameter — you pass the
        # flags yourself.
        "dbt_cmd_flags": ["--defer", "--favor-state"],
        "env": {"DBT_STATE": STATE_DIR},
        "append_env": True,          # WITHOUT THIS, `env` REPLACES the environment
        "full_refresh": False,
    },
    schedule="@daily",
    dag_id="bookshop_slim",
)

Two things in that operator_args block are load-bearing, and both are easy to get wrong in a way that looks fine.

DBT_STATE sets --state, not --defer. It threads the state directory through both phases — Cosmos passes it to the dbt ls at render time and to the dbt run at task time — so the same prior manifest that decides which nodes become tasks is also the baseline available to compare against. But that is only the comparison. Deferral — “when a selected model ref()s an unselected one, resolve it to the production relation instead of building it” — is a separate switch, and Cosmos exposes no defer parameter at all. If you set only DBT_STATE and stop, state:modified+ will correctly prune your task graph, and then every pruned-away parent will simply be missing at run time. You get the smaller DAG and none of the benefit. Pass --defer --favor-state through dbt_cmd_flags and it works. (The equivalent env vars, DBT_DEFER and DBT_FAVOR_STATE, work too if you prefer to keep it all in env.)

append_env: True is not optional. Cosmos defaults it to False, which means the env dict you supply replaces the process environment rather than extending it — so the moment you set DBT_STATE, you also silently unset PATH, HOME, and any credential your profile was reading from the environment. The failure lands somewhere unrelated and confusing. Declare it every time. If you’d rather not run dbt ls on the scheduler at all, the alternative is RenderConfig(load_method=LoadMode.DBT_MANIFEST) pointed at a pre-generated ProjectConfig(manifest_path=...) — but note that renders your project’s own manifest into tasks; it doesn’t do the state comparison for you, so you’d be back to filtering some other way. DBT_LS is what makes state:modified real at render time, and the price of that is the parse-time-on-the-scheduler constraint. Pay it deliberately.

Get the staging right and state:modified+ prunes your task graph down to the changed slice automatically. Get it wrong and dbt either rebuilds the world (no manifest found — dbt treats everything as new) or compares against a stale one, which is worse, because it silently under-builds.

The clone alternative to defer

--defer is elegant for CI, but it has a cost: the deferred models are read straight out of production. Your PR build is now issuing queries against live prod tables. Usually harmless, occasionally not — a heavy PR test suite hammering prod, or a governance rule that CI must never touch production relations, and defer is off the table.

dbt clone is the other answer. It creates zero-copy clones of the models in a manifest into a scratch schema, using the warehouse’s native clone (Snowflake zero-copy clone, BigQuery table clone, and so on — a metadata operation, not a data copy, so it’s fast and nearly free):

# 1. clone the entire prod project into a per-PR scratch schema
dbt clone --state ./prod-artifacts --target ci

# 2. now build only what changed, against the clone
dbt build --select "state:modified+" --state ./prod-artifacts --target ci

After the clone, the scratch schema is a full stand-in for production, so the state:modified+ build finds every upstream ref() already present — no --defer needed, and nothing reads from prod at run time. When does clone beat defer? When you want isolation from production reads, when you’ll run the same CI schema through multiple builds and want a clean warm baseline each time, or when your warehouse’s clone is cheap enough that the up-front clone is a rounding error. Defer wins when clone is expensive or unsupported on your warehouse, and when a read-only touch of prod is acceptable. For the bookshop on DuckDB there’s no meaningful clone primitive, so defer is the natural fit; on Snowflake or BigQuery, clone is very often the better-behaved choice.

Incrementals, full-refresh, and version skew

Two more interactions worth knowing before you trust this in anger.

Deferring into an incremental model. Suppose orders is incremental and your PR edits a downstream mart that ref()s it. With --defer, that ref('orders') resolves to the production orders table — the real, fully-populated incremental table, not an empty scratch copy. That’s exactly right: you get realistic upstream data without rebuilding a large incremental in CI. But watch the override: if your run includes --full-refresh, dbt rebuilds selected incrementals from scratch, and --full-refresh takes precedence over the incremental logic for selected models. Deferred models aren’t selected, so they’re untouched — but the moment an incremental model is in your state:modified+ selection and you pass --full-refresh, it does a complete rebuild in your target schema regardless of its incremental config. In CI that’s often what you want (a clean rebuild of the thing you changed); in a nightly slim run it can be an expensive surprise. Keep --full-refresh out of the routine slim DAG and reach for it deliberately.

Manifest compatibility across versions. The manifest.json schema is versioned and it changes between dbt-core minor releases. A manifest written by dbt 1.11 is not guaranteed to be readable as state by dbt 1.9, and vice versa — dbt will usually warn and may refuse. The practical rule: the dbt version that reads the state manifest should match (or be no older than) the version that wrote it. When you upgrade dbt-core, the first run after the upgrade should regenerate and republish the production manifest before any slim run tries to compare against the old one. Pin the dbt version in your image and treat a dbt upgrade as an event that invalidates the published manifest.

The publish-on-success race. Publish the manifest only when the production run actually succeeded. If a run fails halfway and you still overwrite the published manifest.json, your next slim run compares against a baseline that describes a state production never fully reached — it will under-build, skipping models it thinks are current but aren’t. Make the “upload manifest to object storage” step conditional on the dbt build succeeding (in Airflow terms, a task with trigger_rule="all_success" gated behind the build). And because a build and its upload aren’t atomic, prefer writing to a versioned key or writing-then-copying so a concurrent slim run never reads a half-written file. The manifest is a production artifact now; give it the same care as the data.

Tracing one change through the slim run

It helps to walk a single change end to end, because the moving parts only make sense together. A developer opens a PR that edits orders.sql — adds a total_discount column, nothing else. Here’s what has to line up for the slim build to do the right thing.

First, the production DAG ran last night, built cleanly, and its success-gated upload step pushed target/manifest.json to s3://bookshop-artifacts/dbt/bookshop/prod/manifest.json. That’s the baseline. Second, the manifest-staging step — the small dedicated DAG, or the CI job — has already pulled that object down to include/dbt/state/manifest.json on the machine that will parse and run this build. Third, when the slim DAG is parsed, Cosmos runs dbt ls --select state:modified.body+ --state include/dbt/state and diffs the current project against the baseline. Only orders has a different compiled body, and orders is a leaf, so the + adds nothing downstream: the rendered task graph is a single node.

Now the run. That one orders task executes dbt build --select orders --defer --favor-state --state include/dbt/state. orders does ref('int_order_payments'), int_order_payments isn’t in the selection, so --defer rewrites the reference to production’s int_order_payments, and --favor-state guarantees it uses prod’s copy even if a stale one is loitering in the CI schema. dbt builds exactly one model, reading one deferred upstream table, and the PR gets a real result in the time it takes to rebuild a single mart. If the manifest at any of those three staging points had been stale or missing, this same PR would have either rebuilt the whole project or quietly compared against the wrong baseline — same code, same selector, different outcome, entirely because of where one file was.

Two words spelled the same

One more pass on the vocabulary landmine, because it’s the single most common point of confusion in this whole area. dbt --defer — the subject of this chapter — is about ref() resolution: pointing unbuilt references at production. It has nothing to do with Airflow’s scheduler, workers, or the triggerer. Airflow deferrable operators — an entirely separate feature covered later in the practice series — are about a task releasing its worker slot while it waits, handing the wait to the triggerer process, and resuming later. A Cosmos task running dbt build --defer is not an Airflow deferrable task; it occupies its worker for the whole run. If someone says “the dbt task is deferred,” always ask which “defer” they mean. They are unrelated systems that lost a naming coin-flip.

Final thoughts

state:modified is dbt admitting that most of your project didn’t change since last night, and it’s right — most of it didn’t. The leverage is enormous and the interface is a single selector, which is exactly why it’s tempting to bolt on without the plumbing underneath. Don’t. The whole feature rests on one file being staged to the right place at the right time — and under Cosmos with DBT_LS, “the right time” is parse time on the scheduler, not run time on a worker, which is the detail that turns a working slim DAG into a mysterious one. Version the manifest, publish it only on success, compare against exactly that, and match your dbt version to the one that wrote it. Do that and “only rebuild what changed” is a line of config. Treat the manifest casually and it’s a very confident way to build the wrong thing.

Next: Tests, Retries, and Shipping It

Comments