CI/CD and Deployment Across dbt and Airflow

PR-based Slim CI, cloned Snowflake environments, dbt artifact promotion, DAG delivery, and GitHub Actions shape.

Every other chapter in this series shipped one thing at a time: a dbt model, an Airflow DAG, a warehouse setting. Deployment is where you have to ship them together, because the modern data stack has two codebases that pretend to be one. There’s the dbt project — the models, tests, and snapshots that define what the data means. And there’s the orchestration project — the DAGs, the Cosmos config, the manifest.json that tells Airflow the shape of the dbt graph. A change to the first almost always implies a change to the second, and a CI pipeline that treats them as independent will merrily ship a DAG that points at a manifest describing models that no longer exist.

So this chapter is about two questions that turn out to be the same question. On a pull request: how do you test a dbt change against Snowflake without rebuilding the world or touching production? And on merge: how do the dbt project and its manifest actually reach the Airflow workers, in lockstep with the DAG that runs them? Get both right and a merge to main is a non-event. Get either wrong and you find out at 2am.

The PR problem: test against Snowflake without touching prod

A dbt pull request changes SQL. The only way to know that SQL is correct is to run it against a real Snowflake warehouse — dbt’s tests are selects that have to execute somewhere. That “somewhere” is the whole design problem. It can’t be production ANALYTICS_PROD: a broken PR would corrupt the tables your dashboards read. It can’t be a full rebuild of every model on every push either — on Snowflake that’s a bill, and on a project with hundreds of models it’s a twenty-minute check nobody waits for.

The pattern that solves both is Slim CI: build only the models the PR touched and everything downstream of them, and resolve the untouched upstream models against production instead of rebuilding them. That’s the state:modified+ selector you met in the production chapter, now given a real environment to run in:

dbt build --select state:modified+ --defer --state ./prod-artifacts

Three flags, three jobs. state:modified+ selects the changed models plus their descendants (the trailing +), so a one-line fix to a staging model rebuilds that model and its dependents, not the warehouse. --state ./prod-artifacts points dbt at a directory holding production’s manifest.json — the reference dbt diffs against to decide what “modified” means. And --defer says: for any ref() to a model I’m not building, don’t error and don’t rebuild it — resolve it to the object production already has. That last flag is what lets a PR build the tip of the graph while leaning on prod for the trunk.

Two things have to exist for this to work: a production manifest to diff against, and a Snowflake environment to build into. Take them in order.

Where the production artifacts live, and how they stay fresh

--state ./prod-artifacts is only as good as the manifest sitting in that folder. If it’s a week old, state:modified compares your PR against a week-old graph and either rebuilds too much or misses a change. The manifest has to be the one from the last successful production run, and keeping it current is a job for the deploy pipeline, not a human.

The production job — the one that runs dbt build against ANALYTICS_PROD on merge — writes a fresh target/manifest.json as a side effect of every run. The trick is to publish it somewhere the PR job can fetch it. Two common homes:

  • A GitHub Actions artifact, uploaded by the production job and downloaded by the PR job. Simple, no extra infrastructure, but artifacts are scoped to a workflow run and expire, so you fetch “the manifest from the most recent successful deploy run.”
  • An S3 (or GCS) bucket with a stable key like s3://acme-dbt-artifacts/prod/manifest.json, overwritten on every prod deploy. This is the more durable choice and the one that scales past a single repo — the same object doubles as the manifest Airflow will load, which matters later in this chapter.

Either way the invariant is the same: the production manifest is refreshed on every production run, and the PR job pulls the latest one before it diffs. Miss that and Slim CI silently drifts.

Giving the PR a Snowflake environment: clone vs. defer

Now the environment. There are two ways to give a PR build the production data it needs to resolve those deferred refs, and they represent a genuine trade-off rather than a right answer.

Option one: a dedicated CI database, defer to prod. You keep a permanent ANALYTICS_CI database, build the PR’s state:modified+ models into a PR-scoped schema inside it, and let --defer resolve every unchanged ref() back to ANALYTICS_PROD. Nothing gets cloned; the PR only ever materializes the handful of models it changed. It’s cheap and fast, and it’s the model most teams start with.

The subtlety is a flag called --favor-state. When you defer, dbt’s default is to prefer a relation in your current target if one exists, falling back to the state manifest only when it doesn’t. That default bites in a shared CI database: a previous PR may have left a stale stg_orders sitting in some schema, and dbt will happily resolve to that ghost instead of prod. --favor-state inverts the preference — for any deferred node, use the production relation from --state, even if something by that name exists in the CI database:

dbt build \
  --select state:modified+ \
  --defer --favor-state \
  --state ./prod-artifacts \
  --target ci

Reach for --favor-state whenever your CI database is long-lived and shared, so deferral always means “the real production table,” never “whatever a past run left lying around.”

Option two: a zero-copy clone of production. Snowflake can hand you a full, writable copy of a database in seconds, for no storage cost, because it copies metadata and not bytes:

CREATE DATABASE analytics_ci CLONE analytics_prod;

That single statement gives the PR its own ANALYTICS_CI that is production — every table, every row, every schema — but forked. Cloned objects share prod’s underlying micro-partitions until something writes to them; only the models your PR rebuilds diverge, and only those consume new storage. So the build resolves every ref() against real, complete data that lives right there in the clone. You don’t even strictly need --defer in this mode: the unchanged parents are physically present in analytics_ci, so ref('stg_orders') in the clone just works.

The clone’s virtues are isolation and realism. Writes land in the clone, never in prod, so there’s no shared-schema contamination and no --favor-state footgun. Tests run against a true copy of production’s data volume, which is exactly where the interesting bugs — a join that fans out at scale, an incremental filter that stops filtering — actually surface. The cost is that a clone is a whole database you must remember to destroy.

Which to pick? Defer-to-prod when your prod tables are enormous and cloning a snapshot per PR feels heavy, or when you’re happy for tests to run against production data indirectly. Clone when you want each PR fully sandboxed and don’t mind managing lifecycle. Many teams land on a hybrid: clone prod for isolation, and pass --defer --state ./prod-artifacts so state:modified still has a manifest to diff against — the clone supplies the data, the manifest supplies the “what changed.”

Teardown is not optional

A zero-copy clone costs nothing to create and nearly nothing to hold — until a PR build mutates a few tables and those diverge into real storage, and until you’ve accumulated three hundred abandoned ANALYTICS_CI_<pr> databases because a workflow died before cleanup. Teardown is a first-class step, and it has to run whether the build passed or failed:

DROP DATABASE IF EXISTS analytics_ci;

In practice you give each PR a uniquely named clone — ANALYTICS_CI_PR_1487 — so concurrent PRs never collide, and you drop it in a step that runs if: always(). A nightly sweep that drops any ANALYTICS_CI_PR_% database older than a day is a cheap belt-and-suspenders against the workflow that crashed between clone and drop. The rule: nothing named after a PR should outlive the PR.

The full GitHub Actions workflow

Here’s the PR job assembled — reproducible install, fetch the prod manifest, clone, dbt deps, Slim build, and unconditional teardown:

# .github/workflows/dbt-ci.yml
name: dbt CI
on:
  pull_request:
    paths:
      - "dbt/**"

concurrency:
  group: dbt-ci-${{ github.event.pull_request.number }}
  cancel-in-progress: true

jobs:
  slim-ci:
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: dbt/tpch
    env:
      DBT_PROFILES_DIR: ${{ github.workspace }}/dbt/tpch
      SNOWFLAKE_ACCOUNT: ${{ secrets.SNOWFLAKE_ACCOUNT }}
      SNOWFLAKE_USER: ${{ secrets.SNOWFLAKE_CI_USER }}
      SNOWFLAKE_PRIVATE_KEY: ${{ secrets.SNOWFLAKE_CI_PRIVATE_KEY }}
      CI_DB: ANALYTICS_CI_PR_${{ github.event.pull_request.number }}
    steps:
      - uses: actions/checkout@v4

      - name: Install uv
        uses: astral-sh/setup-uv@v5

      - name: Install dbt (pinned)
        run: uv pip install --system dbt-core==1.11.12 dbt-snowflake==1.11.0

      - name: dbt deps
        run: dbt deps

      - name: Fetch production manifest
        run: |
          mkdir -p prod-artifacts
          aws s3 cp s3://acme-dbt-artifacts/prod/manifest.json \
            prod-artifacts/manifest.json

      - name: Clone production into a CI database
        run: |
          dbt run-operation clone_ci_db --args "{db: ${CI_DB}}" --target prod

      - name: Slim build against the clone
        run: |
          dbt build \
            --select state:modified+ \
            --defer --favor-state \
            --state ./prod-artifacts \
            --target ci \
            --vars "{ci_database: ${CI_DB}}"

      - name: Teardown clone
        if: always()
        run: |
          dbt run-operation drop_ci_db --args "{db: ${CI_DB}}" --target prod

A few things earn their place. dbt deps runs before anything compiles because a project with packages.yml (dbt-utils, codegen, an internal macro package) won’t parse without its dependencies vendored into dbt_packages/ first — skip it and the very next command fails on an unresolved macro. The pinned install (dbt-core==1.11.12, matching the versions this whole series is verified against) means CI runs the exact dbt a developer ran locally, not whatever floated to the top of PyPI this week. The clone and drop are wrapped as dbt run-operation macros so the SQL lives in the dbt project next to everything else, but they’re just the CREATE DATABASE ... CLONE and DROP DATABASE statements above. And concurrency with cancel-in-progress means a second push to the same PR kills the first run — no point cloning prod twice for a branch that just moved.

Authentication is the key pair from earlier in the series, not a password — SNOWFLAKE_CI_PRIVATE_KEY is a GitHub secret, and the CI Snowflake user is a dedicated service account scoped to create and drop ANALYTICS_CI_* databases and read ANALYTICS_PROD, nothing more.

Branch protection: the part that isn’t code

The workflow above is worthless if someone can merge around it. Branch protection on main is what turns a green check from a suggestion into a gate:

  • Require the slim-ci status check to pass before merge.
  • Require the branch to be up to date with main, so state:modified diffs against the manifest for the code you’re actually merging, not a stale base.
  • Require at least one review, and dismiss stale approvals on new commits.
  • Forbid direct pushes to main — every change arrives through a PR that ran CI.

The “up to date” rule is the subtle one for a data stack. If your branch is behind main and someone merged a change to a shared upstream model in the meantime, your Slim CI ran against a manifest that predates their change, and the two modifications can conflict in a way neither PR’s CI ever saw. Requiring the branch to be current forces the diff to reflect reality.

Deploying the dbt project into Airflow

CI is half the story. The other half starts on merge: the dbt project and its manifest.json have to reach the machines that run Airflow. This is the seam beginners miss entirely, because on a laptop the dbt project is just there — Cosmos reads it off the local disk. In production, the scheduler and the workers are containers somewhere else, and the dbt project has to be delivered to them. There are three ways to do it, and they differ mostly in how fast a change propagates.

Baked image. The dbt project is copied into the Airflow Docker image at build time — COPY dbt/ /usr/local/airflow/include/dbt/ in the Dockerfile — and the manifest, generated in CI, is baked in alongside it. When the image deploys, every scheduler and worker gets the exact same project and manifest, atomically. This is the most reproducible option: the image is an immutable, versioned unit, and “which dbt code is running?” has a one-word answer (the image tag). The cost is propagation latency — a dbt-only change means a full image rebuild and rollout, minutes not seconds. For most teams that’s the right trade: data pipelines don’t need sub-minute deploys, and reproducibility is worth more.

Git-sync. A sidecar container clones the dbt repo into a shared volume and pulls on an interval (say every 60 seconds), so the workers always track a branch. Changes propagate fast — merge, and within a pull cycle the workers have the new code, no image rebuild. The catch is that “fast and continuous” is also “no atomic boundary”: mid-pull, some pods may have the new code and some the old, and there’s no single version stamp for a run. Git-sync shines for iterating on DAGs; for the dbt project specifically it reintroduces exactly the freshness problem the baked image solves.

Object-storage (S3) sync. The dbt project and manifest are pushed to S3 by the deploy pipeline, and workers sync them down on a schedule or at task start. It’s a middle ground — more atomic than git-sync if you sync a versioned prefix, faster than a full image rebuild — and it pairs naturally with the S3 manifest you already publish for Slim CI. The same s3://acme-dbt-artifacts/prod/manifest.json that CI diffs against can be the one the scheduler loads.

The manifest-staleness trap under LoadMode.DBT_MANIFEST

Whichever delivery mechanism you choose, there’s a failure mode specific to how Cosmos discovers your graph, and it’s worth stating plainly because it’s the single most common production-Cosmos bug. In production you run LoadMode.DBT_MANIFEST — Cosmos reads a pre-built manifest.json to construct the Airflow tasks, instead of shelling out to dbt ls on every scheduler parse (the Cosmos internals chapter is the deep version of why). That’s the correct, fast choice. But the manifest is a snapshot, and its name is a promise you have to keep:

A stale manifest renders yesterday’s DAG. Add a model, forget to regenerate the manifest, and Cosmos builds a task graph that doesn’t include it — the model silently never runs, and no error fires, because from Airflow’s point of view the task simply doesn’t exist.

The discipline that closes this is to make manifest generation a build step of every deploy, never a thing the scheduler or a human does. In the same pipeline that ships the dbt project, run dbt parse (or dbt compile) against the prod target to write a fresh target/manifest.json, and deliver that manifest with that code:

      - name: Generate a fresh prod manifest
        run: dbt parse --target prod   # writes target/manifest.json, no warehouse queries

      - name: Publish manifest for Airflow + Slim CI
        run: aws s3 cp target/manifest.json s3://acme-dbt-artifacts/prod/manifest.json

dbt parse is the cheapest command that produces a manifest — it builds the graph and stops, touching no warehouse. Because it’s regenerated on every deploy, the manifest can only ever drift if someone changes models without deploying — and if they haven’t deployed, the workers haven’t seen the new code either. The manifest and the project move as one.

Promote the DAG bundle and the dbt project together

Which is the real deployment rule, and it’s boring on purpose: the Airflow DAG bundle and the dbt project are one deployable unit. A Cosmos DAG is not standalone code — it’s a program whose behavior is defined by the manifest it reads. Ship a DAG that expects a model the deployed manifest doesn’t describe, or ship a new manifest to workers still running last week’s DAG file, and you’ve built a version skew that surfaces as a task that mysteriously errors or a model that mysteriously doesn’t run.

The baked image is the cleanest enforcement of this rule precisely because it makes skew impossible: the DAG files, the dbt project, and the manifest are all in one image tag, promoted atomically. If you use git-sync or S3-sync instead, you have to manufacture that atomicity — deploy the manifest and the DAG bundle from the same commit, gate on both syncs completing, and never let one lead the other. A DAG pointing at a manifest from a different commit isn’t a crash; it’s a delayed, confusing failure, which is worse.

Backfilling after a breaking change

There’s one deployment scenario the happy path doesn’t cover, and it’s the one that actually pages you: you merged a change to an incremental model that alters its logic — a new column, a fixed join, a changed grain. As the incremental chapter warned, an incremental table doesn’t retroactively fix itself; the old rows still hold output from the old SQL, because the is_incremental() filter never revisits them. Your deploy succeeded, the DAG is green, and the data is quietly half-old, half-new.

The fix is a --full-refresh, and the coordination is the fiddly part, because you have to reconcile a dbt operation with Airflow’s own rerun machinery. The safe sequence:

  1. Merge and deploy the code first, so the fresh manifest and the new model logic are live on the workers. A full-refresh has to run the new SQL, which means the new code has to be deployed before you trigger it.
  2. Run the full-refresh out of band, scoped to just the affected subgraph, so you rebuild the broken model and its descendants without nuking every incremental table in the project: 
    dbt build --select orders_enriched+ --full-refresh --target prod
  3. Then let Airflow resume normal incremental runs. From the next scheduled run on, is_incremental() is true again against a table that now matches the current logic, so it appends correctly.

In Airflow 3.x, if you’d rather drive the rebuild through the orchestrator than a shell, remember that backfills are now scheduler-managed — you request a backfill (airflow backfill create, or the UI) over a date range and Airflow queues the runs, rather than the old 2.x CLI daemon. The pattern that keeps it clean is a separate, parameterized “maintenance” DAG whose Cosmos build passes --full-refresh, triggered manually for the window you need to repair, while the scheduled DAG stays incremental. Don’t bolt --full-refresh onto the everyday DAG and forget it there — you’ll rebuild all of TPCH from scratch every night and wonder where the credits went. Full-refresh is a scalpel for the day logic changes, not a default.

The ordering is the whole game: code deploys, then data backfills, then the schedule resumes. Fire the backfill before the deploy and it runs the old logic; resume the schedule before the backfill finishes and the incremental run appends onto a half-repaired table. Sequence them and a breaking change becomes a controlled, reviewable operation instead of a corrupted mart you discover three dashboards later.

Final thoughts

Two codebases, one deploy. That’s the sentence this chapter keeps circling back to, because every failure mode here is some version of letting the dbt project and the orchestration project drift apart. Slim CI keeps a PR honest without a full rebuild, but only if the production manifest it diffs against is fresh. A zero-copy clone gives a PR a real Snowflake environment for free, but only if teardown always runs. LoadMode.DBT_MANIFEST makes the scheduler fast, but only if the manifest is regenerated on every deploy. And a Cosmos DAG does what you expect, but only if the manifest, the dbt project, and the DAG file were all promoted from the same commit.

None of these is hard on its own. What makes deployment the discipline it is, is that they’re only correct together — the CI gate, the artifact freshness, the delivery mechanism, and the backfill sequence are one system with one invariant running through all of it: known-good code and known-good artifacts move as a unit, or they don’t move at all. Hold that line and merging to main stops being an act of faith. It becomes the least eventful thing you do all day, which is exactly what a deploy should be.

Next: Lineage, Contracts, and Observability

Comments