How Cosmos Loads and Runs dbt

Load modes, manifests, partial parsing, invocation modes, operator args, and why scheduler parse cost is the hidden price of task-per-model dbt.

By now the bookshop DAG is a pleasing thing to look at. Every model is a cell, every test hangs off its model, and the grid draws the same graph dbt draws. It feels free, because on a five-model project it is free. The trap is assuming that stays true at a hundred models, or three hundred, or the day someone points the same pattern at the analytics monolith with eleven hundred nodes and wonders why the scheduler is suddenly pinned at 100% CPU and the UI takes nine seconds to render a grid.

Cosmos has a hidden cost, and it’s not where beginners look for it. It isn’t in running dbt — that part is the same subprocess dbt always was. The cost is in the fact that Airflow has to know the entire dbt graph before a single task runs, and it has to know it not once, but every time the scheduler re-parses the DAG file. This chapter is about that seam: how Cosmos discovers your dbt graph (LoadMode), how it invokes dbt once tasks are running (InvocationMode), and how the two together decide whether task-per-model scales or collapses.

Two things happen, at two different times

Keep these separated in your head, because conflating them is the root of most Cosmos performance confusion.

Parse time is when Airflow reads your .py DAG file to build the DAG object. This happens constantly — the dag-processor parses every file on a loop (default min_file_process_interval is 30 seconds), and it happens whether or not anything is scheduled to run. When Cosmos builds a DbtDag, it has to enumerate every dbt node at this moment, because the tasks are the DAG. No graph, no tasks.

Run time is when a rendered task actually fires and calls dbt to build one model. This happens on the worker, on the DAG’s schedule.

LoadMode governs parse time — how Cosmos discovers the graph. InvocationMode governs run time — how each task shells into dbt. A slow load mode taxes you 2,880 times a day per file (every 30s), silently, in the scheduler, for a DAG that might run once a night. That asymmetry is the whole story.

The LoadMode menu

Cosmos supports six load modes, set on RenderConfig:

from cosmos import RenderConfig, LoadMode

LoadMode.DBT_LS shells out to dbt ls during parsing. dbt compiles the project in memory, resolves every ref() and source(), and emits the node list Cosmos turns into tasks. This is the most accurate mode — it sees exactly what dbt sees, including anything computed at compile time — and it’s the reason it hurts. dbt ls on the bookshop takes maybe two seconds. On a real project it’s five to fifteen. Multiply by the parse loop and you have a scheduler that spends a meaningful fraction of its life running dbt just to answer “what models exist?”

LoadMode.DBT_MANIFEST reads a pre-built manifest.json — the artifact dbt writes into target/ after any compile — and builds the graph from that file. No subprocess, no compile, just JSON parsing. It’s the fast, calm mode for anything past toy scale. The catch is stated in its name: the manifest is a snapshot. If someone adds a model and you don’t regenerate the manifest, Cosmos renders yesterday’s graph. Freshness becomes your responsibility instead of dbt’s.

LoadMode.CUSTOM parses the project’s .yml and .sql files directly with Cosmos’s own lightweight dbt-graph parser — no dbt executable, no manifest. It’s faster than DBT_LS and needs no artifact, but it’s an approximation: it doesn’t run Jinja, so anything that depends on compile-time logic (macros that generate refs, complex dbt_project.yml var interpolation) can be misread. Good enough for straightforward projects that don’t want to maintain a manifest.

LoadMode.AUTOMATIC is the default, and it’s a heuristic, not a mode of its own. Cosmos looks for a manifest_path on your ProjectConfig; if it finds one, it uses DBT_MANIFEST. If not, and a dbt executable plus profile are available, it falls back to DBT_LS. It’s convenient and it’s exactly why people get bitten: they never set a load mode, ship to production, and inherit DBT_LS’s parse tax without ever choosing it.

Here’s the decision, compressed:

Load modeRuns dbt at parse?AccuracyFreshness burdenUse when
DBT_LSYes, every parseExactNone (always live)Dev, tiny projects, or you truly need compile-time truth
DBT_MANIFESTNoExact (as of build)You keep it freshProduction, anything > ~50 models
CUSTOMNoApproximateNoneNo manifest wanted, simple project
AUTOMATICDependsDependsDependsYou’ve read this table and know what it’ll pick

The one-line rule: development can afford DBT_LS; production should run on DBT_MANIFEST.

Watching the parse tax with your own eyes

You don’t have to take the “seconds × frequency” claim on faith. Airflow tells you how long each file takes to parse. With the bookshop DAG in place on DBT_LS:

astro dev run dags report

Or, more precisely, look at the dag-processor’s own accounting:

astro dev run dag-processor --help   # confirm the processor is running

The number that matters is import time per file, surfaced in the UI under the DAG’s details and in the scheduler logs as parse duration. On a DBT_LS DAG it will track your dbt ls wall-clock almost exactly, because that’s what dominates. Flip the same DAG to DBT_MANIFEST and the parse duration collapses to the milliseconds it takes to read and deserialize a JSON file. That before/after is the entire argument for manifest mode, made concrete on your own hardware.

A related, quieter cost: because DBT_LS invokes dbt in the scheduler, the scheduler’s environment must have dbt installed and be able to reach a target — dbt compiles against the profile. Manifest mode has no such requirement at parse time. The scheduler just reads a file. That decoupling is worth as much as the speed.

Building the manifest in CI, not in the scheduler

If manifest mode is the answer, the operational question becomes: who builds the manifest, and when? The wrong answer is “the scheduler, at parse time” — that’s just DBT_LS wearing a manifest’s clothes. The right answer is CI, at deploy time. The manifest is a build artifact of your dbt project, exactly like a compiled binary, and it should be produced by the same pipeline that ships the code.

The command that produces a manifest without touching your warehouse is dbt parse (or dbt compile, or any command that compiles) — parse is cheapest because it stops after building the graph:

# .github/workflows/deploy.yml (excerpt)
- name: Generate dbt manifest
  working-directory: dbt/bookshop
  run: |
    dbt deps
    dbt parse --target prod    # writes target/manifest.json, no warehouse queries
- name: Stage manifest for the image
  run: |
    mkdir -p include/dbt/bookshop/target
    cp dbt/bookshop/target/manifest.json include/dbt/bookshop/target/manifest.json

Now the manifest rides along in your Airflow image, next to the project. Point Cosmos at it explicitly:

from pathlib import Path
from cosmos import DbtDag, ProjectConfig, RenderConfig, LoadMode

DBT_PROJECT_PATH = Path("/usr/local/airflow/include/dbt/bookshop")

project_config = ProjectConfig(
    dbt_project_path=DBT_PROJECT_PATH,
    manifest_path=DBT_PROJECT_PATH / "target" / "manifest.json",
)

bookshop_dbt = DbtDag(
    dag_id="bookshop_dbt",
    project_config=project_config,
    profile_config=profile_config,   # from chapter 3
    render_config=RenderConfig(load_method=LoadMode.DBT_MANIFEST),
    schedule="@daily",
    start_date=None,
)

Two things make this robust. First, setting manifest_path means AUTOMATIC would also pick manifest mode — but naming load_method=LoadMode.DBT_MANIFEST explicitly removes the guesswork, and a reviewer reading the DAG knows exactly what parse costs. Second, because the manifest is generated with --target prod, it reflects the graph as production compiles it, not whatever a developer’s local vars happened to produce.

The freshness discipline is now a CI invariant, not a human habit: the manifest is regenerated on every deploy, so it can never drift from the deployed code. The only way it goes stale is if someone changes models without deploying — and if they haven’t deployed, the scheduler hasn’t seen the new code either. The two move together.

Partial parsing: dbt’s own cache

There’s a second cache in play, and it belongs to dbt, not Cosmos. When dbt compiles a project it writes a partial_parse.msgpack into target/. On the next invocation, dbt reads that file and only re-parses the files whose timestamps changed, rather than re-Jinja-ing the whole project. On a large project this turns a multi-second compile into a sub-second one.

This matters in two places. If you’re stuck on DBT_LS (dev, or a project where the manifest genuinely can’t be kept fresh), partial parsing is what makes each parse’s dbt ls bearable — it’s the difference between fifteen seconds and three. And when your tasks run dbt at execution time, partial parsing shaves startup off every single model invocation.

Cosmos exposes this through ProjectConfig (not RenderConfig — that’s a TypeError) and operator_args:

project_config = ProjectConfig(
    DBT_PROJECT_PATH,
    partial_parse=True,   # keep target/partial_parse.msgpack between parses (default: True)
)

The gotcha is that partial parsing depends on the partial_parse.msgpack surviving between invocations. In container execution modes, where each task gets a fresh filesystem, it won’t survive unless you mount target/ on a persistent volume — which is exactly the kind of thing that quietly stops helping without any error message. Set partial_parse=False when you specifically want to force a clean parse (CI manifest generation is a fine place to leave it on; a debugging run where you suspect a stale cache is a fine place to turn it off).

InvocationMode: how a running task calls dbt

Load mode is settled. Now the DAG is running and forty tasks need to build forty models. Each one has to get dbt to actually execute. InvocationMode, set on ExecutionConfig, controls how.

from cosmos import ExecutionConfig, InvocationMode

InvocationMode.SUBPROCESS is the classic behavior: each task spawns a fresh dbt run --select the_model subprocess, waits for it, captures its output, and exits. It’s the most isolated option — every model runs in its own process with its own clean interpreter state — and it’s the most robust, because a dbt crash is contained to one subprocess. The cost is startup: every task pays the price of a cold Python interpreter importing dbt and its adapter, which is a meaningful fraction of a second to a second or two, per task. On a graph with hundreds of small models, that fixed overhead starts to dominate the actual SQL.

InvocationMode.DBT_RUNNER uses dbt’s programmatic entry point — dbtRunner from dbt.cli.main — to invoke dbt in-process, inside the Airflow worker’s own Python, no subprocess spawn. dbt and its adapter are imported once and reused. Startup per task drops dramatically because there’s no new interpreter to warm up. This is the faster option and the right default for ExecutionMode.LOCAL on a large graph.

execution_config = ExecutionConfig(
    invocation_mode=InvocationMode.DBT_RUNNER,
)

The tradeoff is exactly the mirror of the speedup. In-process means shared process state. dbt and Airflow now live in the same interpreter, so their dependency trees must be compatible — a version of protobuf or jinja2 that suits one and not the other becomes your problem, whereas a subprocess would have been insulated by its own environment. It also means dbt’s global state isn’t reset between tasks the way a fresh process guarantees, and a hard dbt failure has a slightly larger blast radius. If you’ve isolated dbt with ExecutionMode.VIRTUALENV or a container (next chapter), DBT_RUNNER doesn’t apply the same way — the isolation is the point there, and Cosmos manages invocation accordingly.

The rule of thumb: DBT_RUNNER for speed when Airflow and dbt already share a controlled environment; SUBPROCESS when you want isolation, are debugging a state-leak, or have a dependency conflict you can’t yet resolve. If you leave it unset under LOCAL, recent Cosmos defaults to DBT_RUNNER when the dbt library is importable and falls back to SUBPROCESS otherwise — but, as with load mode, choosing explicitly beats inheriting a default you didn’t read.

operator_args: the flags that ride down to dbt

Invocation mode decides how a task calls dbt; operator_args decides what it says. Every rendered Cosmos task is an operator wrapping a dbt command, and operator_args is the dictionary that flows into all of them — the same way default_args flows into any Airflow task group. This is where the practical dbt flags live.

render_config = RenderConfig(load_method=LoadMode.DBT_MANIFEST)

bookshop_dbt = DbtDag(
    dag_id="bookshop_dbt",
    project_config=project_config,
    profile_config=profile_config,
    render_config=render_config,
    execution_config=execution_config,
    operator_args={
        "vars": {"run_date": "{{ ds }}"},   # dbt --vars, Jinja-templated per run
        "full_refresh": False,               # dbt --full-refresh toggle
        "install_deps": True,                # run `dbt deps` before the model
        # NOTE: "threads" here is a silent no-op — Cosmos filters unknown kwargs.
        # Set it on the profile instead: profile_args={"threads": 4}
        "env": {"DBT_LOG_FORMAT": "json"},   # environment for the dbt process
    },
    schedule="@daily",
    start_date=None,
)

A few of these are worth a sentence each. vars is templated with Airflow’s Jinja context, so {{ ds }} becomes the run’s logical date and dbt sees a real value — this is how you thread Airflow’s sense of “which day” into dbt’s {{ var('run_date') }}. full_refresh=True is the switch you flip for a backfill DAG that must rebuild incrementals from scratch (a whole chapter of its own, later). install_deps=True runs dbt deps before the model so packages are present, which you want when the image doesn’t already carry dbt_packages/; leave it off when deps are baked into the image, since running it per task is wasted work. And threads is dbt’s own concurrency inside a single invocation — mostly moot under task-per-model, because Airflow is already parallelizing across models, but it matters the moment you coarsen granularity and one task builds many models.

The thing to internalize: operator_args is the run-time counterpart to RenderConfig’s parse-time selection. One decides which nodes exist; the other decides what dbt does when each node runs.

The Cosmos cache

Cosmos adds its own caching layer on top of dbt’s. This one is not a config object at all — it’s Airflow configuration, in the [cosmos] section (equivalently AIRFLOW__COSMOS__* environment variables). Reaching for RenderConfig(enable_cache=…) is the obvious guess and a TypeError. When enabled, Cosmos caches the rendered graph — the result of load-mode discovery — keyed on the inputs that would change it (project files, the manifest, relevant config). On the next parse, if nothing that matters has changed, Cosmos serves the cached graph instead of re-running discovery.

# .env  —  or the [cosmos] section of airflow.cfg
AIRFLOW__COSMOS__ENABLE_CACHE=True
AIRFLOW__COSMOS__CACHE_DIR=/usr/local/airflow/include/.cosmos_cache
AIRFLOW__COSMOS__ENABLE_CACHE_DBT_LS=True     # cache the `dbt ls` output specifically

Worth knowing: enable_cache_dbt_ls defaults to True, which means load_via_dbt_ls already checks the cache before shelling out. So the “dbt ls runs on every parse” tax this chapter has been dramatising is, out of the box, smaller than the arithmetic suggests — it runs when the project changes. The tax is real, but the default already blunts it, and you should know that before you go re-architecting around it.

This is most valuable precisely where you’d expect: it blunts the DBT_LS parse tax by not re-shelling dbt ls when the project hasn’t changed since the last parse. It’s less relevant under DBT_MANIFEST, where discovery is already cheap. Point CACHE_DIR at a path that persists between parses (in the Astro image, somewhere under include/ is convenient) and survives restarts if you want the benefit across deploys. And know the escape hatch: if you ever see Cosmos rendering a graph you’re sure is stale, AIRFLOW__COSMOS__ENABLE_CACHE=False is the blunt instrument that forces fresh discovery every parse — useful for a debugging session, wasteful as a permanent setting.

Layered up, that’s three caches doing different jobs: the Cosmos graph cache (skip re-discovery), the manifest (skip compiling to discover), and dbt’s partial parse (skip re-parsing files dbt already knows). They compose. On a well-tuned production DAG, a parse touches none of the expensive paths — it reads a manifest, finds an unchanged graph in the Cosmos cache, and returns in milliseconds.

The overhead wall at 300–500 models

Everything above buys parse-time and startup savings, but there’s a ceiling those savings don’t lift, and it’s structural. Task-per-model means N dbt models become N (or more, counting tests) Airflow tasks. Airflow’s scheduler tracks every task instance — state transitions, dependencies, retries, database rows — and the UI renders every one into the grid. Somewhere around 300 to 500 models, that stops being free regardless of load mode. The scheduler works harder to schedule thousands of task instances per run; the grid, which has to lay out every cell, becomes sluggish to render; the metadata database carries more rows per DAG run. Manifest mode fixes parse cost. It does nothing about the fact that you asked Airflow to individually manage two thousand tasks.

There are two levers, and they trade observability for scale.

Render at the TaskGroup or DAG level instead of per-node. Cosmos’s render_config can control granularity. The extreme is one task that runs dbt build for a whole selection, and you’re back to the naive bridge’s single bar — but you can also split a large project into a handful of DbtTaskGroups along tag or path boundaries, each rendered per-model, so you keep per-model visibility within a domain while capping how much of the graph any one DAG has to draw at once. A thousand-model project sliced into eight domain DAGs is eight manageable grids, not one unmanageable one.

Always pair large graphs with manifest mode. At this scale DBT_LS isn’t a tax, it’s a denial of service on your own scheduler — hundreds of models means a slow compile run thousands of times a day. Manifest mode is not optional past a few hundred nodes; it’s the price of admission.

The honest framing, and the one this whole series keeps returning to: task-per-model is a choice you pay for in scheduler load and grid complexity, and it’s worth it exactly when per-model observability and targeted retries earn their keep. For a 40-model marts DAG that a data team stares at every morning, it’s obviously worth it. For a 900-model project where nobody looks at individual cells and a failure just means “rerun the batch,” rendering all 900 as tasks is buying a feature no one uses at a cost the scheduler pays every 30 seconds. That’s when you coarsen the granularity and let dbt’s own --select do the fine-grained work at run time.

Putting the internals together

A production-shaped configuration for a non-trivial bookshop, with every knob chosen rather than inherited:

from pathlib import Path
from cosmos import (
    DbtDag, ProjectConfig, RenderConfig, ExecutionConfig,
    LoadMode, InvocationMode,
)

DBT_PROJECT_PATH = Path("/usr/local/airflow/include/dbt/bookshop")

project_config = ProjectConfig(
    dbt_project_path=DBT_PROJECT_PATH,
    manifest_path=DBT_PROJECT_PATH / "target" / "manifest.json",
)

render_config = RenderConfig(
    load_method=LoadMode.DBT_MANIFEST,   # no dbt at parse time
    enable_cache=True,                   # skip re-discovery when unchanged
    cache_dir=Path("/usr/local/airflow/include/.cosmos_cache"),
)

execution_config = ExecutionConfig(
    invocation_mode=InvocationMode.DBT_RUNNER,   # in-process, fast startup
)

bookshop_dbt = DbtDag(
    dag_id="bookshop_dbt",
    project_config=project_config,
    profile_config=profile_config,       # from chapter 3
    render_config=render_config,
    execution_config=execution_config,
    schedule="@daily",
    start_date=None,
    default_args={"retries": 2},
)

Read it as a story about when work happens. Parse time reads a manifest that CI built — no dbt, no warehouse, milliseconds. The Cosmos cache short-circuits even that when the graph hasn’t moved. Run time invokes dbt in-process so no task pays for a cold interpreter. Nowhere in the scheduler’s hot loop does anything expensive run. That’s the shape you’re aiming for, and every knob above is there to keep the two clocks — parse and run — from stepping on each other.

Examples in this chapter are docs-checked against the series’ stated versions (Airflow 3.3.0, astronomer-cosmos, dbt-core), but they were not executed in this repository unless a companion project explicitly says so. Cosmos configuration surfaces — LoadMode, InvocationMode, cache and manifest arguments — have moved across Cosmos releases; pin your version and check its docs before shipping.

Final thoughts

The seductive thing about Cosmos is how little it asks on a small project — point it at a folder, get a graph. The dangerous thing is that the defaults that feel free at five models are quietly expensive at five hundred, and the expense hides in the one place engineers rarely think to profile: the DAG parse, running on a loop, in the scheduler, for a job that fires once a day. The fix isn’t clever. It’s separating the two clocks. Discover the graph from a manifest your CI already built, so parse time is a file read. Invoke dbt in-process, so run time isn’t a thousand cold starts. Cache what doesn’t change. And when the graph gets big enough that even a free parse can’t save you from managing two thousand task instances, admit that per-model rendering is a feature with a bill, and coarsen the granularity where nobody’s reading the cells anyway. Cosmos gives you the whole dbt graph in Airflow. What this chapter gives you is the judgment to decide how much of it Airflow should have to hold in its head at once.

Next: Cosmos Execution Modes

Comments