Cosmos: One Airflow Task Per dbt Model

Astronomer Cosmos parses your dbt project and renders every model and test as a real Airflow task, wired in dbt's own dependency order.

The naive bridge left you staring at a single green bar that turns red without telling you which of six models died. The fix isn’t a cleverer BashOperator — it’s teaching Airflow to see the graph dbt already has. That’s the entire job of Astronomer Cosmos: it reads your dbt project, walks the same ref() edges dbt walks, and renders each model as its own Airflow task with its tests hanging off it, all wired in dbt’s dependency order. int_order_payments stops being a line in a log and becomes a cell in the grid — one you can point at, read, and retry on its own.

Installing it

Cosmos is a Python package. It goes in requirements.txt alongside the dbt adapter for your warehouse, because Cosmos runs dbt in the same environment as your Airflow worker.

astronomer-cosmos==1.11.0
dbt-postgres

Pin the version. Cosmos moves fast, and the names in this post — LoadMode, TestBehavior, InvocationMode, the config objects — have all shifted shape at least once across releases. An unpinned astronomer-cosmos in requirements.txt means the next astro dev restart can quietly pull a version whose defaults changed under you, and the first you hear of it is a DAG that parses differently than it did yesterday. Pin the exact version, pin the adapter too if you’re strict about it, and treat a Cosmos bump as a real dependency change you test — not a free upgrade you take by accident.

With the Astro CLI, that’s the whole install — astro dev restart rebuilds the image and both land in the scheduler and workers. There’s no separate dbt Cloud, no sidecar. dbt runs where your tasks run.

The minimal DbtDag

Cosmos needs two things: where your dbt project lives, and how to connect to the warehouse. The first is a ProjectConfig pointing at the project directory. The second is a ProfileConfig, and this is where Cosmos earns its keep — instead of shipping a profiles.yml full of credentials, you hand it an Airflow connection and let it generate the profile at runtime.

from pathlib import Path

from cosmos import DbtDag, ProjectConfig, ProfileConfig, ExecutionConfig
from cosmos.profiles import PostgresUserPasswordProfileMapping

DBT_PROJECT_PATH = Path("/usr/local/airflow/include/dbt/bookshop")

profile_config = ProfileConfig(
    profile_name="bookshop",
    target_name="dev",
    profile_mapping=PostgresUserPasswordProfileMapping(
        conn_id="bookshop_pg",
        profile_args={"schema": "analytics"},
    ),
)

bookshop_dbt = DbtDag(
    dag_id="bookshop_dbt",
    project_config=ProjectConfig(DBT_PROJECT_PATH),
    profile_config=profile_config,
    execution_config=ExecutionConfig(dbt_executable_path="dbt"),
    schedule="@daily",
    start_date=None,
    default_args={"retries": 2},
)

The PostgresUserPasswordProfileMapping takes the bookshop_pg connection you already defined in Airflow — host, user, password, database all come from there — and profile_args layers on the pieces the connection doesn’t carry, like the target schema. dbt never sees a checked-in credential; it sees a profile Cosmos assembled from Airflow’s own secrets backend. Every warehouse adapter Cosmos supports has a matching mapping class, so the pattern is identical whether you’re on Postgres, Snowflake, or BigQuery.

There’s a second way to configure the profile, and it’s the escape hatch worth knowing. Instead of profile_mapping, you can pass profiles_yml_filepath pointing at an existing profiles.yml:

profile_config = ProfileConfig(
    profile_name="bookshop",
    target_name="dev",
    profiles_yml_filepath="/usr/local/airflow/include/dbt/bookshop/profiles.yml",
)

Reach for this when your adapter has no profile mapping — DuckDB is the common one — or when you already maintain a profiles.yml you’d rather not duplicate. The connection mapping is the idiomatic choice because it keeps credentials in Airflow where they belong; the file is the fallback when the mapping doesn’t exist.

That’s a complete DAG. astro dev restart, open the grid, and where the naive bridge showed one task you now see stg_customers, stg_orders, and stg_payments fanning into int_order_payments, which feeds orders and customers — the bookshop graph, drawn in Airflow, with each model’s tests wired in behind it.

How Cosmos learns the graph

Before Cosmos can render a single task, it has to answer one question: what nodes are in this dbt project, and how do they depend on each other? It gets that answer at DAG parse time — the moment the scheduler reads your DAG file to figure out what tasks exist. And crucially, the scheduler re-parses DAG files continuously, on a loop, not just once at deploy. So how Cosmos learns the graph is a decision you make on every parse, forever.

That decision is the LoadMode, set on RenderConfig:

from cosmos import DbtDag, ProjectConfig, RenderConfig, LoadMode

bookshop_dbt = DbtDag(
    dag_id="bookshop_dbt",
    project_config=ProjectConfig(DBT_PROJECT_PATH),
    profile_config=profile_config,
    render_config=RenderConfig(load_method=LoadMode.DBT_LS),
    schedule="@daily",
    start_date=None,
)

There are six load modes, and for a first read the two that matter are:

  • LoadMode.DBT_LS — Cosmos shells out to dbt ls against your project every time the scheduler parses the DAG. This is the most accurate mode: it runs real dbt, so it sees exactly what dbt sees, tags and configs and all. It’s also the most expensive, because “runs real dbt” means dbt parses the entire project — reads every .sql and .yml, resolves every ref() — on the scheduler’s clock, on a loop. For the bookshop’s handful of models that’s milliseconds. For a project with hundreds of models it’s the single thing most likely to make your scheduler sweat.
  • LoadMode.DBT_MANIFEST — Cosmos reads a pre-built manifest.json instead of running dbt at all. The manifest is the artifact dbt writes after it parses a project (via dbt compile, dbt ls, or any run), and it already contains the full node graph. Point Cosmos at one and parsing becomes a file read — no dbt subprocess, no project walk, on the scheduler. The catch is that you now own the manifest: something in your build or deploy has to produce it and put it where the DAG expects it, and a stale manifest renders a stale graph.

The other two round out the set: LoadMode.AUTOMATIC (Cosmos’s default — try the manifest if one is present, otherwise fall back to dbt ls) and LoadMode.CUSTOM (parse the project yourself and hand Cosmos the nodes). AUTOMATIC is a reasonable default precisely because it does the cheap thing when it can.

The parse-cost trade-off — why DBT_LS is calm on a laptop and alarming in production, how partial parsing and manifest generation actually cut the bill, and where the manifest should live in a deploy — is the whole subject of a later chapter on how Cosmos loads and runs dbt. For now the one fact to carry: choosing a load mode is choosing how much work your scheduler does on every parse, and that’s a production concern long before it’s a correctness one.

deps and partial parse

Two RenderConfig knobs sit right next to the load mode because they change what that parse-time dbt ls actually does.

If your project uses packages — anything in packages.yml, like dbt_utils — dbt can’t parse it until those packages are installed, because a missing macro is a parse error. RenderConfig(dbt_deps=True) tells Cosmos to run dbt deps before it lists the project, so the packages are present when dbt ls walks the graph. Leave it on if you use packages; there’s little cost when you don’t.

render_config = RenderConfig(
    load_method=LoadMode.DBT_LS,
    dbt_deps=True,
)

partial_parse is the other one. dbt keeps a partial_parse.msgpack cache of its last parse so it can skip re-reading files that haven’t changed. Cosmos respects it, and it’s on by default — it’s a large part of why DBT_LS stays tolerable across repeated parses. You’d only reach in to disable it (ProjectConfig(partial_parse=False)) when you suspect the cache is stale and want dbt to reparse from scratch. The deep version of this — what the cache buys you and when it lies — lives in chapter 07 too.

From a node to a dbt command

Learning the graph is half the job. The other half is: when a task actually runs, what does it do? Cosmos maps each dbt node to the narrowest dbt command that builds just that node.

A model task runs dbt run --select <model>. A test task runs dbt test --select <model>. A seed runs dbt seed --select <seed>, a snapshot runs dbt snapshot --select <snapshot>. Cosmos passes the --select down so each Airflow task touches exactly one node — that’s the whole reason a failure points at one model instead of the project. Under the hood it’s still dbt; Cosmos is just calling it one node at a time, in the order the graph demands.

Where the tests go is a choice, and it’s the TestBehavior on RenderConfig:

  • TestBehavior.AFTER_EACH (the default) — each model gets a test task wired immediately downstream of it. stg_orders runs, then stg_orders’s tests run, and only if they pass do the models that depend on stg_orders start. This is the setting that catches a bad row before it propagates into int_order_payments, and it’s why the default is what it is.
  • TestBehavior.AFTER_ALL — all the models run first, then a single test task runs every test at the end. Fewer tasks, less structure; a failure tells you a test broke but not that it broke before a downstream model consumed the bad data.
  • TestBehavior.BUILD — instead of separate run and test tasks, Cosmos renders one dbt build --select <node> per node. dbt build runs a model and its tests together as a unit, in dependency order — the same command the naive bridge used, but now per-node instead of whole-project. This is the closest mapping to how dbt itself likes to work.
  • TestBehavior.NONE — no test tasks at all. For when tests run somewhere else entirely.
from cosmos import RenderConfig, TestBehavior

render_config = RenderConfig(test_behavior=TestBehavior.AFTER_EACH)

How the command is invoked

There’s one more layer under “run dbt run --select”: how Cosmos shells out to dbt. That’s the InvocationMode, and there are two.

  • InvocationMode.SUBPROCESS — Cosmos launches a fresh dbt process for the task, exactly as if you’d typed the command. Clean and isolated; every task pays dbt’s startup and project-parse cost again.
  • InvocationMode.DBT_RUNNER — Cosmos calls dbt in-process using dbt’s programmatic dbtRunner, skipping the per-task subprocess spin-up. Faster across many tasks, at the cost of sharing a process.

For the bookshop it genuinely doesn’t matter — pick neither and Cosmos chooses a sensible default. It starts to matter when you have hundreds of tasks each paying subprocess startup, and that — the real per-invocation cost, the version constraints, and when the runner’s shared process bites — is chapter 07’s territory. Here, just know the knob exists and that it’s about how dbt is called, not what it’s told to do.

Every model becomes an Asset

Here’s a Cosmos behavior that’s easy to miss and changes more than you’d expect: by default, Cosmos emits an Airflow Asset for every model it renders. That’s emit_datasets=True, and it’s on unless you turn it off.

An Asset (Airflow 3’s rename of the old Dataset) is Airflow’s handle on “a thing a task produces.” When a model task finishes successfully, Cosmos marks that model’s Asset as updated. The payoff is that other DAGs can schedule on those Assets instead of on a clock:

from airflow.sdk import Asset, dag, task

# a downstream DAG that reacts to the marts being rebuilt,
# not to a fixed time
@dag(
    schedule=[Asset("postgres://bookshop-db.internal:5432/bookshop/analytics/orders")],
    start_date=None,
)
def refresh_orders_dashboard():
    @task
    def rebuild(): ...
    rebuild()

refresh_orders_dashboard()

Now the dashboard refresh runs when the orders model is actually rebuilt — no guessing at a cron time that’s “probably after dbt finishes,” no brittle sensor polling. This is genuinely useful, and it’s the seam that makes cross-tool lineage work later in the series.

It’s also a thing to be aware of before it surprises you. Every model producing an Asset means a large project emits a lot of Assets, all showing up in Airflow’s Assets view, and any of them can become a scheduling trigger someone downstream comes to depend on. If you don’t want that surface — or you’re on a version where the emitted Asset URIs churn between releases and you’d rather not build dependencies on them yet — turn it off:

render_config = RenderConfig(emit_datasets=False)

The default is the right one for most pipelines. Just know it’s on, and that it means finishing a model is an event other DAGs can hear.

The operator_args you’ll actually use

Everything so far configures structure — what tasks exist and how they connect. operator_args configures behavior — the flags Cosmos passes down to the dbt commands the tasks run. It’s a plain dict handed to DbtDag / DbtTaskGroup, and these are the entries worth knowing:

bookshop_dbt = DbtDag(
    dag_id="bookshop_dbt",
    project_config=ProjectConfig(DBT_PROJECT_PATH),
    profile_config=profile_config,
    operator_args={
        "vars": {"start_date": "2026-01-01", "region": "EU"},
        "full_refresh": False,
        "no_version_check": True,
        "install_deps": True,
        # NOTE: "threads" here is a silent no-op — Cosmos filters unknown kwargs.
        # Set it on the profile instead: profile_args={"threads": 4}
        "env": {"DBT_LOG_LEVEL": "info"},
        "append_env": True,                 # WITHOUT THIS, env REPLACES the environment
    },
    schedule="@daily",
    start_date=None,
)
  • vars — dbt’s --vars. This dict is serialized and passed to every model command, so a var('region') in your SQL resolves to "EU". This is your channel for parameterizing a run. (Some Cosmos versions also accept dbt_vars for the same purpose; vars is the one to reach for, and if you’re passing them through a RenderConfig-level path the dbt_vars name shows up there — same values, they merge into the --vars dbt ultimately receives.)
  • full_refresh — dbt’s --full-refresh. Forces incremental models to rebuild from scratch rather than appending. False is the everyday setting; you flip it True for a specific run when an incremental model’s logic changed and yesterday’s rows are wrong.
  • no_version_check — passes --no-version-check so dbt doesn’t abort when the installed dbt version doesn’t exactly match require-dbt-version in dbt_project.yml. Handy when Cosmos and your project disagree on a patch version and you don’t want that to be fatal.
  • install_deps — runs dbt deps at execution time, so packages are installed on the worker before the model runs. This is the run-time sibling of RenderConfig(dbt_deps=True), which handles the parse-time need. Projects with packages usually want both: deps present when Cosmos lists the graph, and deps present when the worker builds a model.
  • env — environment variables for the dbt process: a log level, a feature flag, an API key a macro needs.
  • append_env — travels with env, and you almost always want it True. See below.

Two of these carry traps sharp enough to be worth their own paragraph.

env replaces the environment; it does not add to it. append_env defaults to False, and when it’s false Cosmos passes dbt only the variables in your env dict — the worker’s own environment is not inherited. So the moment you set env to add one log level, you have also silently removed PATH, HOME, and every credential your profile was reading from the environment, and dbt fails somewhere confusing. Set "append_env": True and env behaves the way its name promises. This is one of those defaults that is defensible (reproducibility: the task’s environment is exactly what you declared) and lethal (nobody expects it), so declare it explicitly every time rather than remembering which way it falls.

threads is not a thing. You will see operator_args={"threads": 4} in blog posts, and it does nothing at all — Cosmos filters operator_args against the operator’s real signature and silently drops anything it doesn’t recognise, so there’s no error to tell you. The thread count lives on the profile: ProfileConfig(profile_mapping=…(profile_args={"threads": 4})). Which is a small, sharp instance of the rule this book keeps arriving at: config that looks load-bearing and isn’t is worse than config that’s missing, because you stop looking for the real knob.

The mental model: RenderConfig and LoadMode decide the shape of the DAG at parse time; operator_args decides how each task behaves at run time. Reach for operator_args when you’d have typed a flag after dbt run at the terminal.

When dbt isn’t the whole DAG

DbtDag builds a DAG that is only dbt. That’s the right shape when the dbt run stands alone, but most pipelines don’t — something extracts and loads the raw tables before dbt transforms them, and something consumes the marts after. For that, Cosmos gives you DbtTaskGroup: the same per-model rendering, but as a TaskGroup you drop inside your own @dag, so the dbt graph sits shoulder to shoulder with your other tasks in one picture.

from airflow.sdk import dag, task
from cosmos import DbtTaskGroup, ProjectConfig, ProfileConfig
from cosmos.profiles import PostgresUserPasswordProfileMapping

profile_config = ProfileConfig(
    profile_name="bookshop",
    target_name="dev",
    profile_mapping=PostgresUserPasswordProfileMapping(
        conn_id="bookshop_pg",
        profile_args={"schema": "analytics"},
    ),
)


@dag(schedule="@daily", start_date=None, catchup=False)
def bookshop_pipeline():
    @task
    def extract_load():
        ...  # land this morning's raw rows

    transform = DbtTaskGroup(
        group_id="transform",
        project_config=ProjectConfig("/usr/local/airflow/include/dbt/bookshop"),
        profile_config=profile_config,
        default_args={"retries": 2},
    )

    @task
    def reverse_etl():
        ...  # push marts back to the CRM

    extract_load() >> transform >> reverse_etl()


bookshop_pipeline()

Now extract_load runs, its downstream dbt models unfold as their own tasks inside the transform group, and reverse_etl waits on the whole group before pushing marts back out. The seam between load, transform, and export is one Airflow graph — exactly the “dbt models the warehouse, Airflow models the day” split, made literal. DbtDag and DbtTaskGroup take the same project_config, profile_config, execution_config, and render_config; the only difference is whether dbt is the whole DAG or a movement within it.

What the rendering buys you

Everything the black box swallowed comes back, and a few things you didn’t have before.

Retries are per model. Set default_args={"retries": 2} and it’s int_order_payments that retries when int_order_payments fails — not the staging layer that already passed. The unit of retry is finally the unit of work. Logs are per model too: click the red cell, read that model’s dbt output, done — no scrolling past five models’ worth of compiled SQL to find the one that broke.

And because these are real tasks, dbt’s parallelism is now Airflow’s parallelism: the three staging models run concurrently because Airflow can finally see they don’t depend on each other. The dependency graph you drew in ref() calls is the dependency graph Airflow schedules against.

One thing to be deliberate about is where dbt actually runs. By default Cosmos uses ExecutionMode.LOCAL, which invokes dbt as a subprocess in the worker’s own environment — that’s why dbt-postgres had to be in requirements.txt. It’s the simplest mode and the right default. Cosmos also offers VIRTUALENV (isolate dbt’s dependencies from Airflow’s), DOCKER, and KUBERNETES (run each model in its own container or pod) for when the worker environment gets crowded or you need harder isolation. Those are a later concern — LOCAL is where you start, and for the bookshop it’s all you need.

The caveat that scales

There’s a bill attached to all of this, and it comes due in exactly one dimension: task count.

The bookshop has six models. Rendered with tests AFTER_EACH, that’s maybe a dozen tasks — a grid you can read at a glance, a scheduler that doesn’t notice. Now imagine the real project this pattern gets pointed at: 400 models, each with two or three tests. That’s over a thousand Airflow tasks in one DAG. Every one of them is a row the scheduler tracks, a cell the grid draws, a heartbeat the executor manages. The grid view that made one dbt failure legible at six models becomes a wall you scroll for a minute to find anything. Task scheduling has overhead per task, and a thousand tiny tasks — each running dbt for a few seconds — can spend more wall-clock time being scheduled than being run.

This is the real trade of task-per-model, and it’s worth naming plainly: you buy per-model observability and targeted retries, and you pay scheduler load and grid clutter. For a project of dozens of models it’s an easy trade — the visibility is worth far more than the overhead. For a project of many hundreds, it’s a decision, and the levers you’ll pull are the ones this post introduced: a lighter LoadMode so parsing doesn’t run dbt on a loop, DBT_RUNNER invocation so tasks don’t each pay subprocess startup, and — often — not rendering the whole project at once, but selecting a slice of it. That last one is the whole subject of the next chapter. A dbt project rendered whole is rarely the DAG you actually want to run.

Final thoughts

Cosmos doesn’t run dbt differently; it reveals dbt to Airflow. It learns the graph at parse time, maps each node to the narrowest dbt command that builds it, and hands you back everything the BashOperator hid — per-model logs, per-model retries, real parallelism, an Asset per model that downstream DAGs can wake up on. The parsing rebuilds itself the moment someone adds a model, and the thing you hand-wrote a shell command to bury is now the thing the grid shows you.

The interesting problems start here, not end here. Once every model is a task, two questions open up: how much that costs at scale, and which models you actually want as tasks. The first is a later deep-dive; the second is next — because a dbt project rendered whole is rarely the DAG you want to run.

Next: Selecting What Runs: Tags, Paths, and Freshness

Comments