The Modern Data Stack, Wired Together
Five series in, you have all the pieces of a data platform — this one bolts them into a single working pipeline on Snowflake.
The Modern Data Stack: Snowflake + dbt + Airflow
The capstone — point dbt at Snowflake with key-pair auth, orchestrate it with Airflow and Cosmos, and build a TPCH pipeline end to end.
-
-
Pointing dbt at Snowflake (with Key-Pair Auth)
The dbt-snowflake adapter is one profile away — but in 2026 that profile has to authenticate with a key pair, not a password.
-
Connecting Airflow to Snowflake
Before Cosmos can run dbt on Snowflake, Airflow itself needs a way in — one connection, its own identity, key-pair auth, and a secrets backend that resolves it at runtime.
-
dbt on Snowflake, Orchestrated by Airflow
Cosmos renders your dbt-on-Snowflake project as a task-per-model Airflow DAG — reusing the one connection that already holds your key-pair credentials.
-
Building the TPCH Pipeline
The payoff: a real staging-to-marts dbt project on TPCH, rendered as a Cosmos task group and bracketed by the Snowflake steps only Airflow can do.
-
Incremental Models and Scheduling on Snowflake
Rebuilding all of TPCH every run is a bill, not a feature — incremental models and a schedule that follows the data fix both.
-
Production: The Pipeline That Runs Without You
A green DAG in dev and a pipeline that survives 2am unattended are not the same thing. The difference is idempotency, retries with backoff, alerts that page a human, and a runbook.
-
The Ingestion Layer
How raw TPCH-shaped data actually lands: COPY INTO, Snowpipe, EL tools, loaded-at metadata, and the raw-to-staging contract.
-
Environments and RBAC Across the Stack
Dev and prod databases, dbt schemas, per-developer sandboxes, Snowflake grants, and separate ingest, transform, and reader roles.
-
Warehouses and Cost in Production
Per-workload warehouses, query tags, resource monitors, statement timeouts, and the spend controls a Snowflake/dbt/Airflow stack needs.
-
CI/CD and Deployment Across dbt and Airflow
PR-based Slim CI, cloned Snowflake environments, dbt artifact promotion, DAG delivery, and GitHub Actions shape.
-
Lineage, Contracts, and Observability
The finale: three views of one lineage graph, dbt contracts that let you change a mart on a Friday, and observability that pages for a broken warehouse but not a warned test.