dbt build Is a Black Box in Airflow
dbt already models your pipeline as a graph — running it as one Airflow task throws that graph away right when you need it most.
Orchestrating dbt with Airflow
Turn a dbt project into a real Airflow task graph with Cosmos — a task per model, selectors, state-aware runs, tests, and CI.
-
-
The Naive Bridge: dbt in a BashOperator
The simplest way to run dbt from Airflow is a shell command in a task — build it honestly, because it's the baseline everything better is measured against.
-
Cosmos: One Airflow Task Per dbt Model
Astronomer Cosmos parses your dbt project and renders every model and test as a real Airflow task, wired in dbt's own dependency order.
-
Selecting What Runs: Tags, Paths, and Freshness
One dbt project, many purpose-built DAGs — RenderConfig picks which nodes become Airflow tasks using dbt's own selector syntax.
-
Only Rebuild What Changed
Once the bookshop project grows, rebuilding every model on every run is waste — dbt's state comparison lets you build only what a change actually touched, if you get the manifest plumbing right.
-
Tests, Retries, and Shipping It
A scheduled DAG that runs dbt isn't production yet — production is what you add after it works: legible test failures, warn-severity signals that don't fail loud, surgical retries that won't double-load, real observability off run_results.json, and CI that catches a bad model before it merges.
-
How Cosmos Loads and Runs dbt
Load modes, manifests, partial parsing, invocation modes, operator args, and why scheduler parse cost is the hidden price of task-per-model dbt.
-
Cosmos Execution Modes
LOCAL, VIRTUALENV, DOCKER, KUBERNETES, EKS, ECS, Cloud Run, and the practical question of how code and credentials reach the worker.
-
Assets, Lineage, and the Cross-Tool Seam
How Cosmos emits Airflow assets for dbt models, how data-aware scheduling bridges tools, and where OpenLineage fits.
-
Incremental dbt Under Airflow Backfills
The retry and catchup hazards of stateful dbt models, full-refresh decisions, and data-interval-aware incremental patterns.
-
Alternatives, Secrets, and Multi-Environment dbt
Cosmos versus dbt Cloud jobs and hand-rolled commands, plus target mapping, version pinning, and secrets across dev, CI, and prod.