The Naive Bridge: dbt in a BashOperator

The simplest way to run dbt from Airflow is a shell command in a task — build it honestly, because it's the baseline everything better is measured against.

Before reaching for a framework, build the thing a framework replaces. The simplest way to run dbt from Airflow is to shell out — cd into the project, run dbt build, let Airflow schedule it. It’s a black box, as the last post argued, but it’s a real pattern that plenty of production pipelines run today, and you can’t judge what Cosmos improves until you’ve built the baseline it improves on. So build the baseline, honestly, limits and all. The value isn’t the DAG you end up with; it’s that every rough edge you hit here has a name in the next chapter, and you’ll recognize it because you felt it first.

Getting the project into reach

Airflow needs three things to run dbt: the dbt executable, the bookshop project files, and a profile telling dbt which database to hit.

Local dev here is the Astro CLI — astro dev start spins up the whole Airflow stack — so dependencies go in the project’s requirements.txt:

dbt-core
dbt-postgres

The adapter matters. In the dbt series the bookshop ran on dbt-duckdb, and DuckDB is perfect for a laptop. It’s awkward the moment work distributes across Airflow tasks, because DuckDB is a single local file — one writer, on one machine, with no notion of a connection from elsewhere. Point dbt at Postgres instead. It’s reachable over the network, it’s what an Airflow connection is built to describe, and it survives the jump from your laptop to a worker you don’t control.

The dbt project lives inside the Astro project so the scheduler can see it. Under include/ is the conventional home for files that aren’t DAGs:

include/dbt/bookshop/
  dbt_project.yml
  packages.yml
  models/
    staging/
    intermediate/
    marts/

That packages.yml is easy to overlook and it’s the first thing that will bite you. The bookshop leans on dbt_utils for a few tests and helpers, so the project declares a dependency:

packages:
  - package: dbt-labs/dbt_utils
    version: [">=1.1.0", "<2.0.0"]

On your laptop you ran dbt deps once, months ago, and forgot it existed. The packages are sitting in dbt_packages/ and every dbt build since has found them. A fresh Airflow worker has no such history.

The dependency step the naive DAG forgets

Here is the DAG most people write first, and it’s wrong in a way that passes on your machine and fails in CI:

@task.bash(env={"DBT_PROFILES_DIR": DBT_PROJECT})
def dbt_build() -> str:
    return f"dbt build --project-dir {DBT_PROJECT}"

Run it on a clean worker — a Kubernetes pod that came up thirty seconds ago, or a rebuilt Astro image — and dbt stops before it compiles a single model:

Compilation Error
  dbt found 1 package(s) specified in packages.yml, but only 0
  package(s) installed in dbt_packages. Run "dbt deps" to install...

dbt build does not install packages. It expects them present. On your laptop they are, because you ran dbt deps by hand and the result stuck around. The worker never ran it, so dbt_packages/ is empty, and the run dies at parse time with a compilation error that has nothing to do with your SQL. Any project with a packages.yml breaks this way on a machine that hasn’t been primed.

The fix is one more command, chained ahead of the build:

@task.bash(env={"DBT_PROFILES_DIR": DBT_PROJECT})
def dbt_build() -> str:
    return (
        f"dbt deps --project-dir {DBT_PROJECT} && "
        f"dbt build --project-dir {DBT_PROJECT}"
    )

&& short-circuits: if dbt deps fails to resolve a package — a version pin that no longer exists, a registry that’s down — the whole command exits non-zero and dbt build never runs, which is exactly what you want. You don’t want a build against half-installed dependencies.

You might split this into two tasks, dbt_deps >> dbt_build, for a cleaner grid. It reads better but it’s a trap: dbt deps writes into dbt_packages/ inside the project directory, and as we’re about to see, that directory doesn’t reliably survive from one task to the next. Two tasks means two workers, and the second one may not see what the first installed. Chaining deps and build in a single shell command keeps them on the same machine, in the same filesystem, in the same breath. Keep them together.

Remember this shape, because it doesn’t go away when you adopt Cosmos — it just changes clothes. Cosmos parses your project to build the DAG, and parsing a project with unmet packages fails the same way, so it exposes RenderConfig(dbt_deps=True) to install packages before it renders the graph, plus an operator-level install_deps to install before a run executes. Same problem, same fix, wearing a keyword argument. The naive bridge just makes you see the mechanism bare.

The DAG

The Task SDK’s @task.bash turns a shell command into an Airflow task. Wire it downstream of the extract-and-load that lands the raw tables, so the ordering reflects reality — dbt transforms rows that something else had to fetch first.

from airflow.sdk import dag, task

DBT_PROJECT = "/usr/local/airflow/include/dbt/bookshop"

@dag(schedule="@daily", catchup=False)
def bookshop_transform():

    @task.bash
    def extract_load() -> str:
        return "python /usr/local/airflow/include/el/sync_bookshop.py"

    @task.bash(env={"DBT_PROFILES_DIR": DBT_PROJECT})
    def dbt_build() -> str:
        return (
            f"dbt deps --project-dir {DBT_PROJECT} && "
            f"dbt build --project-dir {DBT_PROJECT}"
        )

    extract_load() >> dbt_build()

bookshop_transform()

--project-dir points dbt at the project; DBT_PROFILES_DIR points it at the profile. The >> makes dbt wait for the load. Nothing surprising — that’s the point. It runs, and every night it’s one task installing packages and doing one dbt build.

@task.bash is the Task SDK’s decorator form, and it’s the idiomatic choice in Airflow 3.x. The classic operator is still there if you prefer instantiating it directly — from airflow.providers.standard.operators.bash import BashOperator (note the providers.standard path; in Airflow 3 the once-core operators live in provider packages). The decorator and the operator produce the same kind of task and hit the same limits, so nothing in this chapter changes if you swap one for the other. I’ll use the decorator; read every claim as true of both.

A word on where dbt thinks it is. Two flags and one environment variable decide that, and their precedence is worth committing to memory because getting it wrong produces baffling “can’t find the project” errors:

  • Project directory — dbt uses --project-dir if you pass it, otherwise the current working directory, otherwise it walks upward looking for a dbt_project.yml. @task.bash runs your command in a temporary working directory, not your project, so relying on cwd is asking for trouble. Pass --project-dir explicitly, or set the decorator’s cwd= argument to the project path — but not both loosely, because a --project-dir that disagrees with cwd will quietly win and confuse the next person reading the log.
  • Profiles directory — dbt uses --profiles-dir if passed, otherwise the DBT_PROFILES_DIR environment variable, otherwise ~/.dbt/. On a worker there is no ~/.dbt/, so if you set neither, dbt looks in the home directory, finds nothing, and errors. Setting DBT_PROFILES_DIR in env= (as above) is the clean way; --profiles-dir on the command line does the same job.
  • Target — --target prod overrides the target: key in your profile, letting one profile carry a dev and a prod output and letting the DAG choose. Handy when the same DAG runs in staging and production against different schemas.

Set the two directories explicitly and you never wonder which project ran against which database — a question you will ask the first time a staging run writes to a production schema because dbt fell back to a default you forgot about.

Credentials without committing credentials

dbt reads database connection details from profiles.yml. The naive version commits host, user, and password straight into the repo, which is fine right up until it’s a breach. Airflow already has a home for secrets — the connection store — so source the profile from there instead of hardcoding it.

The clean trick is to keep profiles.yml free of literal secrets and have it read environment variables, which dbt supports with env_var:

bookshop:
  target: prod
  outputs:
    prod:
      type: postgres
      host: "{{ env_var('DBT_HOST') }}"
      user: "{{ env_var('DBT_USER') }}"
      password: "{{ env_var('DBT_ENV_SECRET_PASSWORD') }}"
      dbname: "{{ env_var('DBT_DBNAME') }}"
      schema: analytics
      port: 5432

Then populate those variables from an Airflow connection at task time, so the credentials live in Airflow’s secret backend and never touch git. Keep them in the task environment, not in the command string. Inline DBT_PASSWORD=... dbt build looks convenient, but command-line arguments are visible to process listings — anyone who can run ps on that worker reads your password in the clear — and they often appear verbatim in Airflow’s rendered-command view, which is exactly the log a teammate opens to debug the task. The environment is still sensitive, but it stays out of the process table and out of the rendered command, and it avoids the easy leak.

from airflow.sdk import dag, task

@task.bash(
    env={
        "DBT_PROFILES_DIR": DBT_PROJECT,
        "DBT_HOST": "{{ conn.bookshop_postgres.host }}",
        "DBT_USER": "{{ conn.bookshop_postgres.login }}",
        "DBT_ENV_SECRET_PASSWORD": "{{ conn.bookshop_postgres.password }}",
        "DBT_DBNAME": "{{ conn.bookshop_postgres.schema }}",
    }
)
def dbt_build() -> str:
    return (
        f"dbt deps --project-dir {DBT_PROJECT} && "
        f"dbt build --project-dir {DBT_PROJECT}"
    )

The DBT_ENV_SECRET_ prefix does one specific thing and it’s worth being precise about, because it’s easy to overtrust: dbt scrubs any environment variable whose name starts with DBT_ENV_SECRET_ from its own output — compiled SQL, the run log, error messages. It does not scrub Airflow’s logs, the process environment, or anything a subprocess dbt spawns might print. It’s a guard against dbt itself echoing the password back at you, not a general secret manager. Keep the credential in Airflow’s backend and out of the command line; treat the prefix as a second layer, not the wall.

dbt names the profile it wants with the profile: key in dbt_project.yml (bookshop here). One connection, defined once in Airflow or in the configured secrets backend, feeds the profile — no plaintext password in the repo, no password on the process command line, and rotating the credential is a change in one place.

The filesystem your task runs on

Everything so far assumed the worker’s disk behaves like your laptop’s. It doesn’t, and the difference is the quiet foundation the rest of this series is built on, so it’s worth making concrete now while the DAG is simple enough to see clearly.

When dbt runs, it writes. Not just to the database — to a target/ directory inside the project, and that directory fills up with artifacts:

include/dbt/bookshop/target/
  compiled/           # your models with Jinja rendered to plain SQL
  run/                # the exact SQL dbt sent to the warehouse
  manifest.json       # the full graph: every node, ref, test, config
  run_results.json    # what happened this run: node, status, timing
  graph.gpickle       # the compiled DAG, dbt's internal form

Two of these matter beyond the current run. manifest.json is the complete picture of your project as of this run — every model, every ref() edge, every test and its config. run_results.json records the outcome of each node: passed, failed, skipped, how long it took. dbt’s state-comparison features — “build only what changed since last time,” “retry only what failed” — read a previous run’s manifest.json and diff against it. That’s the entire mechanism behind dbt build --select state:modified. It needs a manifest from before to compare the manifest from now.

Here’s the problem the naive bridge can’t solve. An Airflow worker’s filesystem is ephemeral. Under the Kubernetes executor, each task is a fresh pod that comes up, runs, and is destroyed — the disk goes with it. Even under a Celery worker that persists, you get no promise the next task or the next day’s run lands on the same worker, or that anything you wrote survives a deploy that rebuilds the image. So the target/ directory dbt so carefully produced exists for the lifetime of one task and then evaporates. Tomorrow night’s run starts with an empty target/ and no memory of tonight.

Which means dbt build --select state:modified has nothing to modify against. There’s no prior manifest on the worker, so dbt can’t tell what changed and either errors or falls back to building everything. The clever selectors that make dbt fast at scale are inert here, not because the flags don’t work but because the state they compare against washed away with the pod. It’s the same reason dbt deps had to run every task — dbt_packages/ is on that same disappearing disk.

You can’t fix this inside a BashOperator without bolting on infrastructure — pushing target/ to S3 or GCS at the end of every run and pulling it back at the start of the next, managing versions, handling the first-run case where there’s nothing to pull. That’s a real pattern and a whole chapter later in this series does exactly that, deliberately, with Airflow object storage. The point for now is to see the gap: dbt’s state model assumes a persistent working directory, and Airflow workers don’t give you one. Every “make dbt incremental in Airflow” story is a story about smuggling target/ across the ephemeral gap. File that away — it’s the exact problem the state-and-deferral chapter opens on.

When dbt fails, what Airflow sees

Run dbt build and dbt does a lot: it runs models, runs tests, and reports each one. When something goes wrong, dbt has a rich, per-node picture of what happened. Airflow, through a BashOperator, sees exactly one number: the process exit code. That gap is the whole reason the next chapter exists, so let’s look straight at it.

dbt’s exit codes are coarse. 0 means everything it attempted succeeded. 1 means at least one node failed — a model errored, or a test failed at error severity. 2 is a usage error, like a bad flag. That’s essentially the vocabulary. A run where twelve models built cleanly and one test failed returns the same 1 as a run where dbt couldn’t connect to the database at all.

The dbt build command interleaves models and their tests in dependency order, and its default failure behavior is a partial one. When a model fails, dbt skips the models downstream of it — they depend on data that isn’t there — but keeps building everything on independent branches. So a single failure in int_order_payments doesn’t stop stg_customers on a parallel branch from completing; it stops orders and customers downstream of it. You end the run with some models built, some failed, some skipped. dbt knows precisely which is which — it’s all in run_results.json — and it returns 1.

Test severity adds a second axis. A dbt test can be declared severity: warn or severity: error (the default). A failing error test fails the run and pushes the exit code to 1. A failing warn test prints a warning and does not fail the run — exit code stays 0. So a dbt build can surface a genuine data-quality problem, tell you about it in the log, and still hand Airflow a green 0. If your definition of “the pipeline is healthy” includes those warnings, the exit code alone will lie to you by omission. (You can promote warnings to failures with --warn-error when you want zero tolerance, but that’s a choice you have to make.)

--fail-fast changes the shape of a failure: instead of continuing to build independent branches after the first error, dbt stops at the first failed node. It’s the difference between “tell me everything that’s broken” and “stop the moment anything is.” Faster feedback and less warehouse spend on a run you already know is doomed, at the cost of a complete failure picture. Useful in CI; a matter of taste in production.

The command choice matters here too. dbt build is deliberate: it interleaves each model with the tests that guard it, so a model’s tests run right after it’s built and before anything downstream depends on it — a failing test stops bad data from propagating, the same as a failing model would. The older habit of dbt run && dbt test splits that into two phases and, in a shell command, two exit codes you’d have to reason about: dbt run can return 0 and then dbt test return 1, and your && chain has to decide whether the run “succeeded.” Worse, run-then-test builds every model first and validates afterward, so bad rows land in downstream tables before any test catches them. dbt build folds the two into one exit code with the right ordering baked in, which is precisely why it’s the command a BashOperator wants — one process, one status, tests where they belong. It doesn’t fix the granularity problem (you still get one number for the whole graph), but it at least makes that one number mean something coherent.

Now stand where Airflow stands. @task.bash launched a subprocess, waited, and read the exit code. It got 1. It has no idea that eleven models passed, that the failure was int_order_payments, that three marts were skipped, or that a warn-level freshness test also tripped. All of that lives in run_results.json on a disk that’s about to vanish. Airflow marks the whole task failed and, on retry, reruns the whole command — dbt deps, then dbt build over the entire project, staging and all — because the project is the only unit it was ever given. There is no model-level granularity to retry, because from Airflow’s side there are no models. There’s a command that returned non-zero.

That is the motivating gap, stated as plainly as it gets: dbt has per-node truth, and the BashOperator throws all of it away at the exit-code boundary. Cosmos’s entire contribution is to stop throwing it away — to make each dbt node its own Airflow task so a failure is a red cell you can name and retry, not a red bar you can only rerun whole. You’ll appreciate that far more having watched a single 1 stand in for everything dbt actually knew.

What you just built, and what it can’t do

It works. It’s scheduled, it’s ordered after the load, its packages install, and its credentials are handled properly. It’s also everything the last post warned about, now concrete.

It’s one task. int_order_payments failing looks identical to stg_orders failing — a red bar and a log to scroll. Retry reruns the whole project, staging and all, because the project is the retry unit. dbt runs stg_customers, stg_orders, and stg_payments in parallel internally, but Airflow sees none of it and can schedule around none of it. A failure halfway through means a full rerun — tolerable if every model is idempotent and cheap, wasteful the day one isn’t. State can’t help you make it incremental, because state washes off the worker between runs. And the exit code that decides pass or fail is a single coarse number that can hide a warning-level problem entirely.

None of that is a bug in the code. It’s the ceiling of the approach. A single shell command can’t expose a graph it treats as an opaque blob, can’t persist the state that would make it incremental, and can’t report at a finer grain than the exit code the shell hands back — no matter how cleanly you wire the connection.

Final thoughts

The BashOperator bridge is worth building precisely because it’s honest about the trade. It’s the least code, the fewest moving parts, and for a small project that reruns cleanly it might be all you ever need. But you’ve now met, in the flesh, the three seams the rest of this series works: the packages that must be installed on every ephemeral worker, the target/ artifacts that vanish before the next run can use them, and the exit code that flattens dbt’s per-model truth to a single bit. The moment you find yourself scrolling logs to learn which model died, or rerunning twenty minutes of staging to retry one broken mart, you’ve hit the ceiling — and that ceiling is the exact shape of the next post. Cosmos doesn’t run dbt differently; it stops hiding dbt’s graph from Airflow.

Next: Cosmos: One Airflow Task Per dbt Model

Comments