Cosmos Execution Modes

Every Cosmos DAG so far has run dbt as a subprocess inside the Airflow worker. That’s ExecutionMode.LOCAL, and it’s been invisible because it’s the default — dbt-postgres sat in requirements.txt, the worker’s Python interpreter shelled out to dbt run, and nobody had to think about where the process actually lived. That works right up until it doesn’t. dbt wants a version of protobuf your Airflow provider pins to something older. A compliance reviewer asks why the process that holds warehouse credentials shares an interpreter with every other operator in the deployment. One transformation needs 16 GB of memory to build while the rest of the pipeline is happy with two.

Each of those is the same question wearing a different hat: where should dbt run, and how much of the worker’s world should it share? ExecutionMode is Cosmos’s answer to that question, and it’s a menu — nine values, from “in the worker, no isolation” to “on managed cloud compute, in its own pod, with its own credentials.” This chapter walks the whole menu, shows how the dbt project and warehouse credentials actually reach a container when dbt no longer runs beside Airflow, and gives you a way to decide when the isolation is worth what it costs. Because it does cost — every step away from LOCAL buys you a boundary and hands you an image to build.

Where the mode goes

Execution mode is set once, on the ExecutionConfig you already pass to every DbtDag or DbtTaskGroup:

from cosmos import ExecutionConfig, ExecutionMode

execution_config = ExecutionConfig(
    execution_mode=ExecutionMode.LOCAL,
    dbt_executable_path="dbt",
)

ExecutionMode.LOCAL is what you get if you never touch it. The rest of this chapter is the argument for the other eight values, and the mechanics that come with each — because the moment dbt stops running in the worker, three things that were free suddenly need arranging: the dbt project has to arrive somewhere the process can read it, the packages have to be installed, and the warehouse credentials have to cross into whatever runtime dbt now lives in.

ExecutionMode has ten values. They sort into four families by where the dbt process ends up.

In the worker, sharing its Python:

LOCAL — dbt runs as a subprocess in the worker’s own environment. It uses the dbt on the worker’s PATH and the adapter you installed in requirements.txt. Zero extra moving parts, and the lowest per-task latency of any mode because there’s no environment to create and no container to schedule. The catch is the shared interpreter: dbt’s dependencies and Airflow’s dependencies resolve to one set of versions, and when they disagree, one of them loses. It’s the right default and the wrong choice the day a version pin collides.

In the worker, but in an isolated Python environment:

VIRTUALENV — Cosmos builds a Python virtualenv, installs dbt and its adapter into it from py_requirements, and runs dbt there. Same machine as the worker, separate dependency tree. This is the fix for the version-collision problem without leaving the worker or building an image. The cost is environment creation time, which virtualenv_dir caching largely amortizes away (more on that below).

In a container or pod, on infrastructure you run:

DOCKER — dbt runs in a Docker container that the worker launches locally, via the Docker daemon. dbt, its adapter, and the project live in the image; the worker just starts it. Hard isolation from the worker’s Python, at the price of building and distributing an image.
KUBERNETES — dbt runs in a Kubernetes Pod that Cosmos launches through the KubernetesPodOperator. Each dbt task (or task group) gets its own pod, scheduled by your cluster, with its own resource requests and limits. This is the workhorse isolation mode for teams already on Kubernetes: per-model compute sizing, a clean security boundary, and reproducibility from a pinned image.

On managed cloud compute:

AWS_EKS — like KUBERNETES, but targeting an Amazon EKS cluster, handling the EKS auth handshake for you.
AWS_ECS — dbt runs as an AWS ECS task (Fargate or EC2-backed). No cluster of your own to run; AWS schedules the container.
GCP_CLOUD_RUN_JOB — dbt runs as a Google Cloud Run Job. Serverless container execution; you pay per run, not for idle capacity.
AZURE_CONTAINER_INSTANCE — dbt runs as an Azure Container Instance. The Azure equivalent of “run this container, bill me for the seconds.”

These four are the same container idea as DOCKER/KUBERNETES, with the scheduling and capacity handed to a cloud provider. You still build the image; you just don’t run the fleet that hosts it.

Deferred to the triggerer:

AIRFLOW_ASYNC — the outlier. Instead of running dbt to completion in a worker slot, Cosmos submits the warehouse-side work and defers to Airflow’s triggerer, freeing the worker while the query runs on the warehouse. It’s built for the case where the expensive thing isn’t dbt’s Python — it’s the SQL executing on a big warehouse — and holding a worker slot for the duration is pure waste. It leans on Airflow’s deferral machinery (the triggerer process from the executors chapter) and currently targets specific adapter paths; treat it as the specialist mode for long warehouse jobs, not a general default.

LOCAL and its ceiling

Start with what LOCAL gets right, because most of the time it’s the correct call and reaching past it is over-engineering.

from cosmos import DbtDag, ProjectConfig, ProfileConfig, ExecutionConfig, ExecutionMode
from cosmos.profiles import PostgresUserPasswordProfileMapping

profile_config = ProfileConfig(
    profile_name="bookshop",
    target_name="dev",
    profile_mapping=PostgresUserPasswordProfileMapping(
        conn_id="bookshop_pg",
        profile_args={"schema": "analytics"},
    ),
)

bookshop_dbt = DbtDag(
    dag_id="bookshop_dbt_local",
    project_config=ProjectConfig("/usr/local/airflow/include/dbt/bookshop"),
    profile_config=profile_config,
    execution_config=ExecutionConfig(execution_mode=ExecutionMode.LOCAL),
    schedule="@daily",
    start_date=None,
)

The project is a directory the worker can already read — /usr/local/airflow/include/dbt/bookshop, baked into the Astro CLI image at build time. The adapter is in requirements.txt. The credentials come from the bookshop_pg Airflow connection, assembled into a profile at runtime by the mapping. Nothing has to be delivered; it’s all present because it’s the same machine and the same image. That co-location is exactly why LOCAL is fast and simple, and exactly what every other mode has to reconstruct by hand.

The ceiling is the shared interpreter. Airflow 3.3 and its providers pin a real set of transitive dependencies, and dbt-core plus your adapter pin their own. When both worlds are compatible — which for a mainstream adapter like dbt-postgres they usually are — LOCAL is frictionless. When they aren’t, you’ll see it at astro dev restart as a pip resolution error, or worse, as a silent downgrade that breaks an operator you weren’t touching. That’s the signal to isolate.

VIRTUALENV: isolation without an image

VIRTUALENV is the smallest possible step off LOCAL. dbt still runs on the worker, but Cosmos builds it a private Python environment first and installs dbt there from a list you provide. Airflow’s dependencies and dbt’s dependencies never have to agree, because they no longer share an interpreter — and you didn’t have to build, push, or pull a container image to get there.

from cosmos import DbtDag, ProjectConfig, ProfileConfig, ExecutionConfig, ExecutionMode
from cosmos.profiles import PostgresUserPasswordProfileMapping

profile_config = ProfileConfig(
    profile_name="bookshop",
    target_name="dev",
    profile_mapping=PostgresUserPasswordProfileMapping(
        conn_id="bookshop_pg",
        profile_args={"schema": "analytics"},
    ),
)

execution_config = ExecutionConfig(
    execution_mode=ExecutionMode.VIRTUALENV,
    virtualenv_dir="/usr/local/airflow/dbt_venv",
)

bookshop_dbt = DbtDag(
    dag_id="bookshop_dbt_venv",
    project_config=ProjectConfig("/usr/local/airflow/include/dbt/bookshop"),
    profile_config=profile_config,
    execution_config=execution_config,
    operator_args={
        "py_requirements": [
            "dbt-postgres==1.9.0",
            "dbt-core==1.9.4",
        ],
        "install_deps": True,
    },
    schedule="@daily",
    start_date=None,
)

Two arguments carry the mode. py_requirements is the pip list Cosmos installs into the virtualenv — this is where you pin the exact dbt-core and adapter versions you want, independent of whatever Airflow’s own environment resolved to. install_deps=True tells Cosmos to run dbt deps so the project’s packages.yml dependencies (dbt_utils and friends) land too. With LOCAL you’d have installed all of this in requirements.txt, mixed in with Airflow’s; here it’s a separate, pinned tree that can’t perturb the worker.

The one thing to understand about VIRTUALENV is the cost model, because it’s the difference between a tolerable mode and a slow one. Building a virtualenv and pip-installing dbt is not free — do it fresh for every task and you’ve added tens of seconds of setup to each model. virtualenv_dir is the fix: point it at a persistent path and Cosmos caches the environment there, building it once and reusing it across tasks and runs. Without virtualenv_dir, Cosmos creates an ephemeral environment per task invocation — correct, but wasteful. With it, the first task pays the build cost and the rest run against the cached env. On a shared worker, make sure that path is on a filesystem all workers can reach (or accept a per-worker cache); the cache is only a speedup if the worker that runs the task can see it.

VIRTUALENV is the right answer to a dependency problem: dbt and Airflow won’t cohabit, but you don’t need a security boundary or per-model compute. It keeps everything on the worker — same logs, same retries, same locality — and only quarantines the Python. When the problem is bigger than dependencies, you go to containers.

How the project and credentials reach a container

Here’s the pivot. Under LOCAL and VIRTUALENV, the dbt project is a directory on the worker and the credentials come from an Airflow connection the worker can read. Move to DOCKER, KUBERNETES, or any cloud mode, and dbt runs in a container that is not the worker — a separate filesystem, a separate process, possibly a separate machine. Two things that were ambient now have to be delivered on purpose: the code and the credentials.

Getting the project in. There are two established patterns, and the choice shapes your whole deployment story.

Bake it into the image. Your dbt project is copied into the container image at build time, so /dbt/bookshop is just there when the pod starts. This is the reproducible option: the image tag pins both the dbt version and the exact project code, so a run is a function of an immutable artifact. The cost is a build-and-push step in CI every time a model changes — the image is the unit of deployment, and stale images run stale SQL. This is the default recommendation for KUBERNETES and the cloud modes, because reproducibility is usually why you left LOCAL.
Mount or git-sync it. The project lives outside the image — on a mounted volume, or pulled fresh by a git-sync init container that clones your dbt repo into a shared volume before the dbt container starts. Now code changes don’t need an image rebuild; the pod pulls the latest on launch. The trade is that a run is no longer a pure function of one artifact — it depends on the image and whatever the volume/branch held at launch. Good for fast iteration, weaker on “prove exactly what ran.”

Getting the credentials in. The container can’t read your Airflow connections directly, so the profile has to travel as environment variables or mounted secrets. Cosmos still generates the profile from your ProfileConfig — the mapping resolves the Airflow connection at task time — and then the container modes pass the resulting values into the container as env vars (through the underlying operator’s environment/env_vars) or reference a mounted secret. The rule that held in the LOCAL chapters still holds: credentials live in Airflow’s secrets backend, never in the image. You bake the code into the image; you inject the secrets at launch. An image with a warehouse password compiled in is a leaked password with a version tag.

Keep those two axes straight — project delivery (bake vs mount) and credential delivery (env vs mounted secret) — and every container mode is a variation on the same shape.

KUBERNETES: a pod per dbt task

KUBERNETES is the mode most teams reach for when LOCAL and VIRTUALENV aren’t enough, so it’s worth a full worked example. Cosmos renders each dbt node into a KubernetesPodOperator task: the model runs in its own pod, scheduled by your cluster, with resources you specify and an image you pin.

from cosmos import (
    DbtDag,
    ProjectConfig,
    ProfileConfig,
    ExecutionConfig,
    ExecutionMode,
    RenderConfig,
)
from cosmos.constants import TestBehavior

# The dbt project is baked into this image at /dbt/bookshop, along with
# dbt-core and dbt-postgres. Built and pushed by CI on every model change.
DBT_IMAGE = "registry.example.com/bookshop-dbt:2026.06.24"

profile_config = ProfileConfig(
    profile_name="bookshop",
    target_name="prod",
    # No profile_mapping here: the profile is assembled inside the pod from
    # env vars we inject below, so we point at a profiles.yml baked into the
    # image that reads those env vars via dbt's env_var() Jinja.
    profiles_yml_filepath="/dbt/bookshop/profiles.yml",
)

bookshop_dbt = DbtDag(
    dag_id="bookshop_dbt_k8s",
    project_config=ProjectConfig(
        # The project already lives in the image; this path is inside the pod.
        dbt_project_path="/dbt/bookshop",
    ),
    profile_config=profile_config,
    execution_config=ExecutionConfig(execution_mode=ExecutionMode.KUBERNETES),
    render_config=RenderConfig(test_behavior=TestBehavior.AFTER_EACH),
    operator_args={
        "image": DBT_IMAGE,
        "namespace": "airflow",
        "get_logs": True,
        "on_finish_action": "delete_pod",
        # Warehouse credentials injected from Airflow's secrets backend as
        # env vars; profiles.yml inside the image reads them with env_var().
        "env_vars": {
            "DBT_PG_HOST": "{{ conn.bookshop_pg.host }}",
            "DBT_PG_USER": "{{ conn.bookshop_pg.login }}",
            "DBT_PG_PASSWORD": "{{ conn.bookshop_pg.password }}",
            "DBT_PG_DBNAME": "{{ conn.bookshop_pg.schema }}",
        },
        # Per-model compute sizing lives here.
        "container_resources": {
            "requests": {"cpu": "500m", "memory": "1Gi"},
            "limits": {"cpu": "2", "memory": "4Gi"},
        },
    },
    schedule="@daily",
    start_date=None,
)

Read that top to bottom and you can see all four axes resolved. Project delivery: baked into DBT_IMAGE at /dbt/bookshop, pinned by the 2026.06.24 tag, so the run is reproducible from an immutable artifact. Credential delivery: the env_vars block pulls host, user, password, and database out of the bookshop_pg Airflow connection using template expressions, and hands them to the pod as environment variables; the profiles.yml inside the image reads them with dbt’s env_var() — the password is never in the image, only in Airflow’s backend. Compute sizing: container_resources sets requests and limits, so a heavy mart model can ask for 4 GB while staging models stay small. Pod hygiene: get_logs=True streams the pod’s dbt output back into the Airflow task log so the grid still shows per-model logs, and on_finish_action="delete_pod" cleans up the pod when the task finishes so you don’t leak completed pods across the cluster. (You will see is_delete_operator_pod=True in older examples. Don’t copy it: the current KubernetesPodOperator still accepts that kwarg but no longer reads it — the behaviour is driven entirely by on_finish_action, whose default happens to be delete_pod. Setting the old flag is a no-op that looks load-bearing, which is the worst kind of dead config. Use on_finish_action, whose other values — keep_pod, delete_succeeded_pod — are what you actually want when you need a failed pod left behind to autopsy.)

Notice what changed versus LOCAL beyond the mode value: the profile now comes from a baked profiles.yml reading env vars, because the pod has no access to Airflow’s connection store the way a LOCAL subprocess does — Cosmos resolves the connection in the worker while templating, then ships the values into the pod. That indirection is the whole “how do credentials cross the boundary” story in one block.

AWS_EKS, AWS_ECS, GCP_CLOUD_RUN_JOB, and AZURE_CONTAINER_INSTANCE are the same skeleton with the scheduling handed off. AWS_EKS adds the EKS cluster-auth handshake to the KUBERNETES pattern; ECS swaps the pod for a Fargate task definition; the GCP and Azure modes launch a Cloud Run Job or Container Instance respectively. In each, operator_args carries the provider-specific knobs (task definition, cluster name, region, service account) instead of namespace/container_resources, but the two questions — how does the project arrive, how do the credentials arrive — have the identical two answers.

What isolation does to logs, artifacts, and retries

There’s a subtler tax on the container modes that doesn’t show up in the config: three things that were automatic under LOCAL become design decisions you have to make on purpose.

Logs. Under LOCAL, dbt’s stdout is the worker’s stdout, so it lands in the Airflow task log for free — click the cell, read the model’s output. Under KUBERNETES the output is happening in the pod, and it only reaches the Airflow task log if the operator streams it back. That’s what get_logs=True bought in the example above; drop it and the pod runs fine but the grid shows you an empty log and you’re back to kubectl logs to see why a model failed. The per-model log visibility that made Cosmos worth it in the first place is a flag you have to remember to set.

Assets do not survive the jump to a container. This is the sharpest edge in the whole chapter, and nothing warns you about it: Cosmos’s asset emission lives on the local operator base class. DbtKubernetesBaseOperator and DbtDockerBaseOperator don’t inherit it, so under KUBERNETES or DOCKER no assets are emitted at all — and every downstream DAG that does schedule=[Asset(...)] (the entire cross-tool seam of chapter 09) simply stops firing. No error, no warning; the consumer just never runs again. If data-aware scheduling is load-bearing for you, that is an argument for LOCAL or VIRTUALENV that outweighs most of the isolation benefits below.

Artifacts. dbt writes run_results.json, manifest.json, and compiled SQL into its target/ directory — and you cannot read any of it afterwards, in any mode. This surprises people who assume LOCAL means “on the worker, where I left it”: Cosmos copies the project into a tempfile.TemporaryDirectory(), runs dbt inside it, and deletes the directory when the task ends. target/ goes with it. Containers make this more obvious (the pod is gone too) but not more true. If anything downstream depends on those artifacts — a docs build, a freshness check, an external lineage push — you have to ship them out deliberately, and Cosmos gives you the switch: AIRFLOW__COSMOS__REMOTE_TARGET_PATH (plus _CONN_ID) uploads target/ to object storage after each invocation. Set it once and the artifacts outlive the task in every mode.

Retries. The retry mechanics still work — default_args={"retries": 2} retries the failed task in every mode — but what a retry costs changes. A LOCAL retry re-runs a subprocess: near-instant. A KUBERNETES retry re-schedules a pod: it pulls the image (unless cached on the node), starts the container, and re-establishes the warehouse connection before dbt even begins. On a cold node that’s tens of seconds of overhead per attempt. It’s rarely a dealbreaker, but it means “just bump retries to 5” is a cheaper decision under LOCAL than under a container mode, and flaky-test retries you’d shrug at locally become visible latency in a pod-per-model DAG.

None of this argues against isolation. It argues for going in with eyes open: the boundary you gained is real, and so is the plumbing — log streaming, artifact export, retry cost — that the boundary made your problem instead of the worker’s.

A decision framework

The temptation, once you know the menu exists, is to reach for isolation because it feels more serious. Resist it. Every mode past LOCAL adds latency, moving parts, or an image to keep current, and none of that pays for itself unless you have the problem it solves. Pick the mode from the problem, not the prestige.

Stay on LOCAL when the worker image is already yours to control, the dbt adapter cohabits cleanly with your Airflow providers, and the project is a size where one worker’s resources are plenty. This is most bookshop-scale pipelines. LOCAL is faster and simpler, and simpler is a feature in the thing that runs at 6am unattended.

Reach for VIRTUALENV when your problem is purely dependency conflict — dbt wants versions Airflow won’t allow — but you don’t need a security boundary or per-model compute. It’s the cheapest isolation: no image, no registry, no cluster, just a pinned py_requirements and a cached virtualenv_dir. If the only reason you’re eyeing containers is a pip resolution error, VIRTUALENV probably ends the conversation.

Reach for KUBERNETES (or DOCKER, or a cloud mode) when at least one of these is a real, named requirement:

Security boundary. The dbt process holds warehouse credentials, and policy says it can’t share an interpreter or a host with the rest of the deployment. A pod with its own service account and its own injected secret is an auditable boundary; a subprocess in the worker isn’t.
Per-model compute sizing. One transformation needs far more memory or CPU than the rest, and you’d rather size that one pod than provision every worker for the worst case. container_resources per task is exactly this lever.
Reproducibility. You need a run to be a function of an immutable, tagged artifact — the same image produces the same dbt version and the same SQL, every time, forever. A baked image tag delivers that in a way a mutable worker environment can’t.
Blast-radius isolation. A runaway dbt process shouldn’t be able to starve or crash the worker that’s also running your sensors and your other DAGs. A pod that OOMs takes down a pod.

Reach for AIRFLOW_ASYNC when the expensive part is the warehouse query, not dbt’s Python, and the runs are long enough that holding a worker slot for the duration is measurable waste. It’s a specialist tool; if you’re asking “should I use async mode,” the answer is usually no unless you have long-running warehouse jobs and a triggerer sized for them.

The honest default for a team that already runs on Kubernetes and needs any of those four requirements is KUBERNETES. The honest default for everyone else is LOCAL, with VIRTUALENV as the one-step escape hatch when dependencies collide. Everything in between is a specific answer to a specific need — and if you can’t name the need, you want LOCAL.

Examples in this chapter are docs-checked against the series’ stated versions, but they were not executed in this repository unless a companion project explicitly says so.

Final thoughts

Execution mode is a single ExecutionConfig argument, but it decides your whole operational posture: how fast tasks start, how your image pipeline works, where credentials cross a boundary, and what happens when one model misbehaves. The nine values line up on one axis — how much of the worker’s world dbt is allowed to share — from LOCAL (all of it) through VIRTUALENV (everything but the Python) to the container and cloud modes (nothing but what you inject). Once dbt leaves the worker, two ambient facts become deliberate deliveries: the project has to be baked or mounted, and the credentials have to be injected as env or secret, never compiled into the image.

Choose from the problem. A dependency clash is a VIRTUALENV. A security boundary, per-model sizing, or a reproducibility mandate is KUBERNETES or a cloud mode. A long warehouse query that’s wasting a worker slot is AIRFLOW_ASYNC. And a controlled worker image running a mainstream adapter is LOCAL — the mode you should have to be argued out of, not into. The isolation is real, and so is its cost; spend it where it buys you something.

Next: Assets, Lineage, and the Cross-Tool Seam