dbt build Is a Black Box in Airflow

dbt already models your pipeline as a graph — running it as one Airflow task throws that graph away right when you need it most.

dbt hands you a DAG. Every ref() you write draws an edge, and by the time the bookshop project has stg_customers, stg_orders, stg_payments, an int_order_payments in the middle, and orders and customers at the marts layer, dbt knows the exact order those models have to run and which ones can run in parallel. Airflow hands you a different DAG — one that schedules work, retries what fails, and stitches together tasks across every tool in your stack. Both are directed acyclic graphs. Both are good at what they do. The temptation is to bolt them together the obvious way, and the obvious way quietly discards half of what you’re paying for.

The obvious way is one task that runs dbt build. It works. It’s also a black box.

One green bar, no information

Picture the bookshop pipeline running nightly in Airflow as a single dbt build task. Most nights it’s a green bar in the grid and nobody thinks about it. Then int_order_payments throws a division-by-zero on a day when a customer has a refund larger than their order total, and the whole thing goes red.

Now look at what Airflow can actually tell you. One task failed. That’s it. Which of the six models died? Open the logs, scroll past the compiled SQL for every model that ran before it, and read dbt’s own output to find out. Airflow — the thing whose entire job is to make failures legible — has no idea int_order_payments exists. It saw a shell command exit non-zero.

Retrying is worse. The failure was in one intermediate model, but your retry button reruns dbt build from the top. The three staging models that already succeeded run again. If you’re paying per query, you pay again. If a staging model takes twenty minutes, you wait again. The unit of retry is the entire project, because the entire project is one task.

And dbt’s parallelism is stuck behind Airflow’s back. dbt can run stg_customers, stg_orders, and stg_payments at the same time — it knows they don’t depend on each other. But Airflow can’t schedule around a graph it can’t see. It can’t slot a dbt model between an unrelated extract and a downstream export, can’t give one slow model its own pool, can’t show you the lineage. dbt’s structure is real, and inside a single task it’s invisible.

Two kinds of parallelism, and why they’re not the same

There’s a line people repeat about this — “with a task per model, dbt’s parallelism becomes Airflow’s parallelism” — and it’s true enough to be useful and wrong enough to bite you. The two systems parallelize at completely different granularities, against completely different limits, and pretending they’re one number is how you end up either starving the warehouse or hammering it.

dbt’s parallelism is threads, set per target in profiles.yml. It’s in-process: a single dbt build invocation walks the graph and, whenever more than one model is unblocked, hands them to a thread pool inside that one process. threads: 4 means at most four models compile-and-submit-and-wait concurrently, all from one machine, one dbt process, one warehouse connection pool. The number is really a statement about the warehouse — how many concurrent queries you’re willing to throw at Postgres or Snowflake before you saturate a virtual warehouse or blow through a connection limit. It has nothing to do with how much compute the orchestrator has.

Airflow’s parallelism is task slots, and it’s spread across workers. How many tasks run at once is governed by a stack of ceilings: the executor’s worker capacity, max_active_tasks_per_dag on the DAG, the global parallelism, and — the knob that actually matters here — pools. A pool is a named bucket of slots that any set of tasks can be assigned to; put every dbt-model task in a pool with four slots and no more than four run at a time, no matter how many workers are idle. That number is a statement about the cluster — how many task processes your workers can host.

So when you fan a dbt project out into one task per model, you don’t inherit dbt’s threads setting. You retire it. Each model now runs in its own dbt run --select one_model invocation (this is roughly what Cosmos does, and post seven pulls the covers off it), so each task is a single-threaded dbt process — threads inside it is nearly moot because there’s only ever one model to run. The concurrency that used to be dbt’s thread pool is now Airflow’s pool of slots. The catch is that these two limits protect different things, and the migration silently swaps one for the other:

  • dbt threads were protecting the warehouse from too many simultaneous queries.
  • Airflow slots are protecting the cluster from too many simultaneous task processes.

If you set threads: 4 for years to keep a small Snowflake warehouse happy, then fan out to a task per model on a beefy Kubernetes cluster with no pool limit, Airflow will cheerfully launch fifteen model tasks at once and fifteen queries land on a warehouse tuned for four. Nothing in the fan-out carried your old ceiling across. You have to re-express it as a pool. The good news is that a pool is a better tool for the job — it’s visible in the UI, it can be shared across DAGs so your nightly bookshop run and an ad-hoc backfill compete for the same warehouse budget, and it decouples “how much warehouse can I use” from “how much cluster do I have.” But it’s a knob you now own explicitly, in Airflow, and “dbt’s parallelism becomes Airflow’s” is doing a lot of quiet work to gloss that hand-off.

What good looks like

The fix isn’t to run dbt differently. It’s to make Airflow see the same graph dbt sees. One Airflow task per dbt model. One task per test. int_order_payments becomes a node in the Airflow grid, sitting downstream of the three staging models and upstream of orders, exactly where dbt already knows it belongs.

Everything the black box swallowed comes back. A failure points at one node — you read the model name off the grid, not out of a log. A retry reruns that model and its downstream, not the staging layer that already passed. The concurrency moves under Airflow’s control, where a pool can cap warehouse load and a scheduler can interleave a dbt model with an unrelated extract. Lineage shows up where you’d look for it.

Translating a dbt project into Airflow tasks by hand would be miserable and would rot the moment someone adds a model. So you don’t do it by hand. Astronomer Cosmos parses your dbt project and renders its graph as Airflow tasks for you, and it’s what this series builds toward in post three. First, though, it’s worth building the black box on purpose — the naive bridge is a real pattern, and you can’t appreciate what Cosmos buys you until you’ve felt what it costs.

When the black box actually wins

Here’s the part the “task per model” pitch usually skips: sometimes the monolith is the right call, and not only for toy projects. There’s a cost model underneath this, and it’s U-shaped — the fan-out pays off in the middle and stops paying at both ends.

At the small end, the black box wins because there’s almost nothing to make legible. A project with eight models that runs in ninety seconds and reruns cleanly from the top gains little from per-model tasks. When it fails, dbt build’s own output tells you which model died in the first screen of logs — you didn’t need the grid to find it. A full rerun costs ninety seconds and a handful of cheap queries, so the “retry reruns everything” objection is a rounding error. Against that, a task per model adds real overhead: every task is a separately scheduled unit with its own queue time, its own worker startup, its own row in the metadata database. For eight models the ceremony costs more than it saves. One @task.bash running dbt build is the correct engineering answer, and reaching for Cosmos here is over-tooling.

At the large end — and this is the counterintuitive one — the fan-out can lose again, for the opposite reason. Per-task overhead is roughly fixed per task, so it scales linearly with model count, while the benefit (finding the one model that failed, retrying only its subtree) is realized rarely, only on the failing runs. Two costs grow with the size of the project:

  • Scheduler and slot overhead. Every model task is scheduled, queued, dispatched to a worker, heartbeated, and recorded. A few hundred milliseconds to a couple of seconds of pure orchestration tax per task sounds trivial until you multiply it by 800 models and 800 tests. A monolithic dbt build that finishes in twelve minutes can turn into a run that spends more wall-clock time in Airflow’s scheduling machinery than in the warehouse, because you’ve handed the scheduler 1,600 things to babysit instead of one.
  • The parse tax. To render a dbt project as tasks, something has to know the graph before the DAG runs — which means running dbt ls (or parsing manifest.json) at DAG-parse time. Cosmos does this for you, but it isn’t free, and on a large project it isn’t cheap. If it happens on every scheduler parse loop, a big project can add seconds to minutes of dbt invocation to the scheduler itself, on a cadence measured in tens of seconds. This is exactly why the render side of Cosmos has a whole configuration surface for caching the manifest and avoiding re-parsing — a surface post seven is about, and a tax you don’t pay at all with one dbt build.

Put rough numbers on it. Say the orchestration tax is a conservative one second per task — schedule, queue, dispatch, heartbeat, record. A 12-model project fanned into models plus tests is maybe 24 tasks: 24 seconds of pure tax against a run that’s minutes long. Invisible. Scale that to an 800-model project with a test on nearly every model — call it 1,500 tasks — and you’ve added roughly 25 minutes of orchestration overhead, some of it serialized behind slot limits, to a dbt build that finished in twelve. The warehouse work didn’t change; you spent the extra half hour inside Airflow. Add the parse tax on top: if rendering that project runs dbt ls on the scheduler every parse loop and that call takes 30 seconds on a big project, the scheduler is now doing 30 seconds of dbt work on a repeating cadence just to draw the DAG, whether or not it runs. Neither number exists for a single dbt build. The fan-out didn’t make the pipeline slower to transform — it made it more expensive to orchestrate, and past a certain size that bill is real.

So the decision boundary isn’t “big project → fan out.” It’s shaped by three questions:

  1. How often does it fail in a way where finding the failed model is the bottleneck? If failures are rare and dbt’s own logs already point at the culprit, per-model legibility buys little.
  2. How expensive is a full rerun? If any single model is slow or costly, the ability to retry one subtree instead of the whole project is where the fan-out earns its keep — this is the strongest single reason to do it.
  3. What does the parse/schedule overhead cost at your model count? Below a hundred-ish models it’s noise; into the many hundreds it’s a line item you must actively manage.

The sweet spot for a task per model is the messy middle — dozens to low hundreds of models, some of them slow or expensive, failing often enough that “which model, and rerun just that branch” is a question you ask for real. The bookshop lives squarely in that middle, which is why this series orchestrates it that way. But if you run a 900-model monolith that almost never fails and reruns in fifteen minutes, keeping it as one task — or fanning out only the expensive marts and leaving staging as a single build — can be the more honest answer. Cosmos supports exactly that middle ground too, via how you group and select, which is post four’s subject. The point is that “always fan out” is a slogan, not a cost model.

The integration landscape

“Run dbt from Airflow” has more than one answer, and Cosmos is a choice among them, not the only door. Four options come up in practice, and they sit on a line from least to most structure:

A plain BashOperator (or @task.bash). One task, dbt build, the black box we’ve been describing. Least code, fewest dependencies, no knowledge of dbt’s graph. It’s the right tool at the small end and the honest baseline everything else is measured against — which is why the next post builds it deliberately before we replace it.

A hand-rolled manifest parser. dbt writes a manifest.json describing every node and edge. You can read it yourself, walk the nodes, and generate one Airflow task per model with dbt run --select calls wired up by dependency:

import json
from airflow.sdk import dag, task

with open("/usr/local/airflow/include/dbt/bookshop/target/manifest.json") as f:
    manifest = json.load(f)

models = {
    uid: node for uid, node in manifest["nodes"].items()
    if node["resource_type"] == "model"
}
# then: create a task per model, and set task_a >> task_b
# for every parent/child edge in node["depends_on"]["nodes"]

This is genuinely a few dozen lines to a first working version — and then it’s a project. You have to handle tests, snapshots, and seeds; decide how to select subsets; keep the manifest fresh; render it without re-parsing on every scheduler loop; deal with dbt’s full-refresh and vars; and maintain all of it as dbt’s manifest schema evolves version to version. Cosmos is, in essence, this parser, done well, by people who track dbt’s manifest schema so you don’t. Building your own is reasonable if you have needs Cosmos doesn’t meet; for most teams it’s re-implementing a maintained library.

Astronomer Cosmos. DbtDag renders an entire dbt project as an Airflow DAG, and DbtTaskGroup embeds a dbt project as a group of tasks inside a larger DAG — the second is what you reach for when dbt is one stage among extract, load, and reverse-ETL. Cosmos parses the project, builds the task-per-model graph, threads your Airflow connection into a dbt profile, and gives you selectors, execution modes, and test placement as configuration. It runs dbt on your own workers, against your own warehouse. This is the series’ spine, starting in post three.

DbtCloudRunJobOperator. If your team runs dbt Cloud, the transformation already lives there — jobs, scheduling, logs, and its own scheduler. In that world Airflow’s job isn’t to run dbt model-by-model; it’s to trigger a dbt Cloud job and wait for it:

from airflow.providers.dbt.cloud.operators.dbt import DbtCloudRunJobOperator

run_bookshop = DbtCloudRunJobOperator(
    task_id="run_bookshop",
    dbt_cloud_conn_id="dbt_cloud",
    job_id=12345,
    wait_for_termination=True,
)

Note what this is: a black box by design. From Airflow’s side it’s one task — the whole dbt run collapses back into a single node, because the per-model graph lives inside dbt Cloud’s UI, not Airflow’s. That’s a fine trade when dbt Cloud is where your analytics engineers already work and you want Airflow only to own the seam — fire the dbt Cloud job after the load, block until it’s done, then kick off reverse-ETL. You give up in-Airflow lineage and subtree retries; you gain a managed dbt runtime and a clean separation of concerns. The choice between this and Cosmos is really a choice about where the dbt graph should be legible — inside Airflow (Cosmos) or inside dbt Cloud (the operator).

Why Airflow at all, and the seam it owns

If dbt already has a graph and already knows the order to run it in, why not just dbt build on a cron and skip Airflow entirely? Because dbt’s graph stops at the edge of the warehouse. It knows int_order_payments sits downstream of stg_orders and stg_payments. It has no idea that all three depend on a Fivetran sync landing this morning’s rows, or that the finance team’s export can’t run until orders is fresh.

Airflow owns the seams between tools. The extract-and-load that feeds the bookshop’s raw tables, the dbt run that transforms them, the reverse-ETL that pushes marts back into the CRM — those live in different systems, and something has to sequence them, retry the flaky HTTP call in the middle, and backfill a two-week gap when a source was quietly broken. That something is Airflow. dbt models the warehouse; Airflow models the day.

In Airflow 3.x, the mechanism that actually stitches those seams together has a name: Assets. (If you learned Airflow on 2.x, these are Datasets, renamed — same idea, sharper API.) An Asset is a named piece of data that a task can declare it produces, and a DAG can declare it consumes by scheduling on it. Instead of guessing a cron time that’s “probably after Fivetran finishes,” the dbt DAG says schedule=[Asset("bookshop_raw")] and runs the instant the load marks that Asset updated — data-aware scheduling, not clock-aware guessing:

from airflow.sdk import Asset, dag

@dag(schedule=[Asset("bookshop_raw")], catchup=False)
def bookshop_transform():
    ...

This is the through-line of the whole Fivetran → dbt → reverse-ETL seam. The load task produces the bookshop_raw Asset; the dbt stage consumes it and, model by model, produces an Asset per model — Cosmos does this automatically when you turn on emit_datasets, so every dbt model becomes a first-class Asset in Airflow’s lineage; and the reverse-ETL export schedules on the marts Assets it cares about, running only when orders is genuinely fresh. Three tools, three schedules, and none of them is a cron guess — each stage is triggered by the data the previous one produced. That’s Airflow owning the day at the granularity of individual tables, and it’s the payoff that a single dbt build on a timer can’t reach. The assets-and-lineage chapter later in the series builds this seam end to end; for now it’s enough to know that “Airflow owns the seams” isn’t a metaphor — it’s a concrete schedule=[Asset(...)] and a per-model Asset that Cosmos emits for you.

Final thoughts

The mistake isn’t running dbt from Airflow — that’s the right instinct. The mistake is flattening dbt’s graph into a single node on the way in by default, without asking whether flattening is what you want. Most of the time, in the messy middle where the bookshop lives, it isn’t: you want failures that point at one model, retries that rerun one subtree, concurrency you cap with a pool, and seams stitched by Assets rather than crontab arithmetic. Sometimes — a tiny project, or a giant one that reruns cleanly and never fails interestingly — the black box is the correct, cheaper answer, and reaching for Cosmos is over-engineering. A good integration doesn’t hide one tool’s structure from the other; it lets both graphs be one graph when that’s worth paying for. The rest of this series is about getting there deliberately — but the honest first step is to build the black box and stare at it.

Next: The Naive Bridge: dbt in a BashOperator

Comments