Selecting What Runs: Tags, Paths, and Freshness

A DbtDag with no filtering renders every model in the project, every time. For the bookshop that’s five models and their tests — fine. But projects grow, and “rebuild the entire warehouse” is almost never the DAG you actually want. You want a nightly job that touches only the models tagged for nightly. You want a marts-only DAG that assumes staging already ran. You want to rebuild one report and everything feeding it, and nothing else. Cosmos gives you all three from the same dbt project, and it does it with a syntax you already know: dbt’s own node selectors.

RenderConfig is the filter

The knob is RenderConfig, and its two arguments — select and exclude — decide which dbt nodes turn into Airflow tasks before the DAG is even built. This is a render-time filter, not a runtime one: unselected models never become tasks, so they never show up in the grid at all. What you select is what you see.

The selector strings are dbt’s, verbatim. Three flavors cover almost everything:

• Tag — "tag:nightly" selects every model carrying that tag in its config.
• Path — "path:models/marts" selects everything under a directory.
• Graph operators — "+orders" selects orders and all its ancestors; "orders+" selects orders and everything downstream of it.

Mix them freely, because that’s what dbt does. A list means “the union of these selectors,” the same as passing multiple --select arguments on the command line.

There’s a second reason render-time selection matters, beyond a tidy grid: cost. Cosmos parses your dbt project to build the DAG — by default by running dbt ls against the project — and it does that parse every time the scheduler reprocesses the DAG file. A select narrows what Cosmos has to reason about, but more importantly it keeps a five-hundred-model project from rendering five hundred tasks into a DAG that only ever needed forty. Fewer rendered nodes means a smaller DAG for the scheduler to serialize, schedule, and draw. On a large project the difference between “render everything and let the run figure it out” and “render the slice this DAG exists for” is felt in parse time and UI responsiveness, not just aesthetics. (Chapter 7 gets into caching that parse so it doesn’t rerun on every heartbeat; for now, the point is that select is doing structural work, not cosmetic.)

Three bookshop DAGs from one project

Start with a nightly DAG. Tag the models that belong in the nightly run — say stg_orders, stg_payments, int_order_payments, and orders all carry tags: ['nightly'] in their config — and render only those:

from cosmos import DbtDag, ProjectConfig, RenderConfig

nightly = DbtDag(
    dag_id="bookshop_nightly",
    project_config=ProjectConfig(DBT_PROJECT_PATH),
    profile_config=profile_config,
    render_config=RenderConfig(select=["tag:nightly"]),
    schedule="0 2 * * *",
    start_date=None,
)

The grid now holds exactly the nightly models — the customer-facing marts the finance team reads at 2 a.m. get built, and nothing you tagged as weekly-only clutters the picture.

A marts-only DAG is a path selector. When your staging layer is loaded by a separate upstream job, you don’t want to re-render it here:

render_config=RenderConfig(select=["path:models/marts"])

Now orders and customers are the only tasks — the DAG assumes their inputs are already fresh and just rebuilds the presentation layer.

And building one mart plus its lineage is what the graph operators are for. To rebuild orders and everything it depends on — int_order_payments and the three staging models beneath it — lead with +:

render_config=RenderConfig(select=["+orders"])

That single selector walks the dependency graph backward and renders the full subtree feeding orders, in order. Same project, same profile_config, a different slice — and exclude is the mirror image when it’s easier to say what you don’t want: exclude=["tag:deprecated"] keeps everything but the models you’re sunsetting.

The rest of the selector algebra

The three flavors above get you a long way, but dbt’s selector language is bigger than that, and every bit of it works inside RenderConfig because Cosmos hands the strings straight to dbt’s graph parser. It’s worth learning the whole grammar, because the difference between the DAG you meant and the DAG you got usually comes down to one operator.

Union versus intersection. This is the distinction that trips people up first, and it’s worth being pedantic about because getting it backwards silently gives you the wrong DAG. dbt’s rule is:

• A space is a union. tag:nightly tag:finance selects everything nightly or finance. Multiple space-separated arguments on the CLI, and a Python list here, mean the same thing.
• A comma is an intersection. tag:nightly,tag:finance selects only models that are nightly and finance.

# union — nightly OR finance
render_config=RenderConfig(select=["tag:nightly", "tag:finance"])
render_config=RenderConfig(select=["tag:nightly tag:finance"])     # same thing

# intersection — nightly AND finance
render_config=RenderConfig(select=["tag:nightly,tag:finance"])

The comma is the one that surprises people, because everywhere else in computing a comma separates items in a list — here it joins conditions. Say it out loud once and it sticks: space widens, comma narrows. So ["path:models/marts,tag:nightly"] is “nightly models under marts,” while ["path:models/marts", "tag:nightly"] is “everything under marts, plus everything tagged nightly” — two very different DAGs from one character.

If you’re ever unsure, don’t reason about it: run dbt ls --select "<your selector>" against the project and count the nodes. That is exactly what Cosmos is about to do on the scheduler, so what dbt ls prints is precisely what your DAG will contain.

Depth-limited graph operators. Bare + walks the graph as far as it goes. Prefix or suffix it with a number to cap the hops. "2+orders" selects orders plus two layers of ancestors — int_order_payments and the staging models directly beneath it, but no further. "orders+3" selects orders and three layers of descendants. On the bookshop the graph is shallow enough that 2+orders and +orders land on the same set, but on a real project with a ten-deep lineage the depth limit is how you rebuild “this model and its immediate feeders” without dragging in the entire raw layer. The number goes on the side the arrow points from: N+model limits ancestors, model+N limits descendants.

The @ operator. "@orders" is the one people forget exists, and it’s the correct choice more often than you’d think. It selects orders, everything downstream of it, and all the ancestors of those downstream nodes. In plain terms: everything you’d need to fully rebuild orders and everyone who depends on it, including the sideways inputs those dependents also read. If a shared staging model feeds both orders and customers, @orders pulls in customers’s other parents too, so the downstream rebuild has all its inputs. +orders+ would miss those sideways ancestors; @orders catches them.

It helps to resolve these against a concrete graph. The bookshop’s lineage is stg_customers, stg_orders, stg_payments → int_order_payments → orders, with stg_customers also feeding customers directly. Against that:

• "+orders" → stg_orders, stg_payments, stg_customers, int_order_payments, orders. The full ancestry.
• "orders+" → just orders (nothing is downstream of it).
• "1+orders" → int_order_payments, orders. One hop back.
• "+int_order_payments+" → the three staging models, int_order_payments, and orders — the intermediate model with its full family in both directions.
• "@orders" → same as +orders here, but the moment a dashboard exposure hangs off orders, @orders would also drag in every other input that exposure’s models need. On the shallow bookshop the operators overlap; the differences show up the instant the graph branches, which is exactly when you can’t afford to guess.

Print the resolved set before you trust a selector on a real project — dbt ls --select "@orders" lists the nodes a selector matches without running anything, and it’s the fastest way to confirm a DAG will render what you think it will.

Method selectors beyond tag and path. tag: and path: are two methods out of a family. The ones worth knowing:

• config:materialized:incremental — select by any node config, so you can render only your incremental models, or only the ones with a particular meta key.
• resource_type:seed — select by node type: model, seed, snapshot, test, source, exposure. A seed-only DAG is select=["resource_type:seed"].
• source:raw — select models by the source they read (source:raw.orders narrows to one table); pairs naturally with the freshness gate below.
• test_type:singular (or generic, unit) — select tests by kind, so a data-quality DAG can render just the singular tests without the models.
• exposure:finance_dashboard — select everything feeding a named exposure, which is how you rebuild exactly what a downstream BI dashboard depends on.

All of them compose with the union/intersection and graph rules. select=["source:raw+ tag:nightly"] is “nightly models that descend from any raw source” — a source method, a graph operator, and a tag intersection in one string.

Named selectors in selectors.yml

Once a selector string grows past about two operators, it stops being readable and starts being a thing you copy-paste between DAGs and get subtly wrong. dbt’s answer is selectors.yml at the project root: you name a selection once, define its logic in structured YAML, and refer to it by name everywhere.

selectors:
  - name: nightly_marts
    description: Finance-facing marts and their lineage, nightly cadence.
    definition:
      intersection:
        - tag: nightly
        - union:
            - method: path
              value: models/marts
            - method: config
              value: materialized:table

That defines “nightly-tagged nodes that are either under models/marts or materialized as tables.” The YAML form spells out the union/intersection nesting that the string grammar packs into spaces and commas, which is exactly the point — the gnarlier the logic, the more the structured form earns its keep. In RenderConfig you reference it by name — and note it is not a select= value. There is no selector: method in dbt’s grammar (try it and you get Runtime Error: 'selector' is not a valid method name). RenderConfig has a dedicated field, which maps to dbt’s --selector flag:

render_config=RenderConfig(selector="nightly_marts")     # not select=["selector:..."]

Now every DAG that wants the nightly-marts slice references one definition. Change the cadence rules in selectors.yml and every DAG that uses selector:nightly_marts re-renders against the new logic on the next parse — no hunting through Python files for a selector string you inlined three times. This is the same instinct as a dbt macro: name the thing, define it once, call it by name.

When select and exclude both apply

You can set select and exclude together, and the order of operations matters. dbt resolves select first to get the candidate set, then removes anything matching exclude from it. exclude always wins the tie — a node caught by both is dropped.

render_config=RenderConfig(
    select=["+orders"],
    exclude=["tag:deprecated"],
)

That renders the full orders lineage except any node in it tagged deprecated. If a staging model feeding orders is on its way out, this builds everything around it and skips it — useful during a migration when the replacement isn’t wired in yet. The mental model: select draws the circle, exclude cuts holes in it, and you can’t select your way back into a hole. When a DAG renders fewer tasks than you expected, an over-broad exclude shadowing part of your select is the first thing to check.

Where the tests land

RenderConfig has one more argument worth setting deliberately: test_behavior. By default Cosmos renders each model’s tests as their own tasks that run immediately after that model — stg_orders builds, stg_orders’s tests run, and only then does anything downstream proceed. That’s TestBehavior.AFTER_EACH, and it’s the strict, fail-fast choice: a bad stg_orders never contaminates int_order_payments, because the test gate stops the branch cold.

The alternative is TestBehavior.AFTER_ALL, which builds every model first and runs all the tests together at the end. That’s fewer task boundaries and a bit less scheduling overhead, at the cost of letting a broken model’s rows flow downstream before any test catches them. The trade is legibility and containment versus speed: AFTER_EACH tells you precisely which model’s data failed and quarantines it there; AFTER_ALL finishes faster but hands you a pile of failures to sort through after the fact.

There’s a third option that changes the plumbing underneath, not just the timing. TestBehavior.BUILD renders each node with dbt build instead of the dbt run + dbt test pair that AFTER_EACH uses. The distinction is subtle but real. Under AFTER_EACH, a model’s task is a run and its tests are separate test tasks that Cosmos wires as downstream nodes — two task boundaries, and the test task selects that model’s tests explicitly. Under BUILD, each resource is a single build task, and dbt build interleaves runs and tests in dependency order within dbt itself: it builds a model, tests it, and refuses to build a downstream model whose upstream tests failed — all inside one command invocation. It also covers seeds and snapshots in the same pass, which run/test don’t. Reach for BUILD when you want dbt’s own build-order guarantees (a failed test blocks descendants at the dbt level, not just the Airflow level) and you’re happy with coarser-grained tasks; reach for AFTER_EACH when you want each test surfaced as its own retryable grid cell. And there’s TestBehavior.NONE to skip test rendering entirely — useful when tests live in their own DAG.

One detail that bites people: which tests actually attach to a selected model. When you select orders and Cosmos renders its tests, dbt has to decide whether a test that touches both orders and some un-selected model comes along. That’s test_indirect_selection, and Cosmos exposes it on ExecutionConfig — not on RenderConfig, which is the natural guess and a TypeError:

from cosmos.constants import TestIndirectSelection

render_config    = RenderConfig(select=["orders"])
execution_config = ExecutionConfig(
    test_indirect_selection=TestIndirectSelection.CAUTIOUS,   # default: EAGER
)

The split makes sense once you see it: RenderConfig decides which nodes become tasks, ExecutionConfig decides how dbt is invoked — and indirect selection is a flag on the dbt test command, not a graph question.

The three modes decide the edge cases. EAGER (dbt’s default) runs a test if any model it references is selected — so a relationships test between orders and an unselected customers runs anyway. That’s usually what you want, but it means a test can run against a parent that this DAG didn’t build. CAUTIOUS runs a test only if all the models it references are selected — the orders/customers relationships test sits out unless both are in the DAG, which keeps a marts-only run from testing against staging it never touched. BUILDABLE is the middle ground: it runs a test if all referenced models are either selected or buildable from the selection’s ancestors — looser than cautious, tighter than eager. For the bookshop’s marts-only DAG, CAUTIOUS is the honest choice: don’t assert a foreign-key relationship against a table this DAG didn’t build and can’t vouch for.

Gating on fresh sources

Selecting the right models is only half the question. The other half is whether the raw data those models read is fresh enough to bother running at all — and dbt answers that with source freshness, a check Cosmos can put right at the front of the DAG.

If the bookshop’s raw orders table hasn’t been touched since yesterday’s Fivetran sync, running the nightly transform just recomputes the same numbers and burns warehouse credits doing it. dbt’s freshness check compares a timestamp against the thresholds you declare on the source:

sources:
  - name: raw
    tables:
      - name: orders
        loaded_at_field: _synced_at
        freshness:
          warn_after: {count: 12, period: hour}
          error_after: {count: 24, period: hour}
          filter: _synced_at > dateadd('day', -3, current_timestamp)

Two thresholds, two outcomes — and the mechanism is simpler than most people assume. dbt source freshness exits 0 when everything is fresh and also when a source merely warns; it exits 1 when a source crosses its error_after line. That’s it. There is no third exit code, and Cosmos does not special-case anything.

That’s worth sitting with, because it means the behavior everyone attributes to Cosmos is really just dbt’s. A freshness warning leaves the task green not because Cosmos decided to be lenient, but because dbt itself returned 0 — a warn is dbt saying “getting old, not dead yet,” and it reports that in its output rather than in its exit status. Only an error_after breach exits non-zero and fails the task. So if raw.orders crosses error_after the freshness task errors, the DAG halts, and you get paged about a broken source — not a mysteriously empty orders mart three tasks later.

But a warn shouldn’t vanish silently either — twelve hours stale is worth a Slack ping even if it doesn’t stop the run. That’s what on_warning_callback is for. Cosmos lets you hand the source-rendering path a callable that fires when freshness (or a test, under some behaviors) returns a warning rather than an error:

def warn_slack(context):
    send_slack_message(
        channel="#data-alerts",
        text=f"Source freshness warning in {context['task_instance'].task_id}",
    )

render_config = RenderConfig(
    source_rendering_behavior=SourceRenderingBehavior.ALL,
    on_warning_callback=warn_slack,
)

Now a warn keeps the DAG green and posts to #data-alerts, and an error still fails hard. You get the two-tier signal dbt designed — a nudge and an alarm — mapped onto Airflow instead of flattened into a single pass/fail.

A word on loaded_at_field. It names a column dbt reads to find the newest row — _synced_at here, whatever your loader stamps. Omit it and modern dbt falls back to warehouse metadata freshness: it asks the warehouse itself when the table was last modified (via the platform’s metadata tables) instead of running a max() over a column. Metadata freshness is cheaper — no scan — and it’s the right call when your loader doesn’t write a reliable timestamp, but it measures when the table changed, not when the data is timestamped for, which can differ if rows arrive late. Prefer loaded_at_field when you have a trustworthy column; lean on metadata freshness when you don’t. The filter: line narrows what freshness even looks at — here, only rows from the last three days — which keeps the freshness query fast on a huge append-only table and stops ancient backfilled rows from skewing the “newest row” calculation.

Cosmos doesn’t render sources as tasks by default — you opt in through RenderConfig:

from cosmos.constants import SourceRenderingBehavior

render_config = RenderConfig(
    source_rendering_behavior=SourceRenderingBehavior.ALL,
)

With ALL (or WITH_TESTS_OR_FRESHNESS, which renders a source only when it carries tests or a freshness: block), any source with thresholds renders as its own task that runs dbt source freshness ahead of the models — so the gate shows up in the grid as a real, retryable node instead of a check buried inside a monolithic build. Fresh data, and the transform proceeds; stale data, and you’ve caught the real problem at the door instead of debugging its symptom downstream. It’s the same instinct as the whole series: make the failure legible at the point it actually happened.

Putting it together

Everything in this chapter composes into one purpose-built DAG. Here’s the nightly-marts job as it actually wants to be: rendered from a named selector, gating on source freshness with a Slack callback on warnings, testing cautiously so it never asserts against tables it didn’t build, and building each node with dbt build so failed tests block descendants at the dbt level.

from cosmos import DbtDag, ProjectConfig, RenderConfig
from cosmos.constants import (
    SourceRenderingBehavior,
    TestBehavior,
    TestIndirectSelection,
)


def warn_slack(context):
    send_slack_message(
        channel="#data-alerts",
        text=f"Freshness warning: {context['task_instance'].task_id}",
    )


nightly = DbtDag(
    dag_id="bookshop_nightly_marts",
    project_config=ProjectConfig(DBT_PROJECT_PATH),
    profile_config=profile_config,
    render_config=RenderConfig(
        select=["selector:nightly_marts"],
        source_rendering_behavior=SourceRenderingBehavior.WITH_TESTS_OR_FRESHNESS,
        test_behavior=TestBehavior.BUILD,
        test_indirect_selection=TestIndirectSelection.CAUTIOUS,
        on_warning_callback=warn_slack,
    ),
    schedule="0 2 * * *",
    start_date=None,
)

Read it top to bottom and it’s the whole chapter in one object: select names the slice, source_rendering_behavior puts the freshness gate at the front, test_behavior and test_indirect_selection decide how the slice checks itself, and on_warning_callback wires the two-tier signal. Change the definition in selectors.yml and this DAG re-renders against it. Point a second DbtDag at a different selector and you have a second slice — same project, same profile, no duplication. The DAG is thin because all the real decisions live in the config.

Final thoughts

The shift here is small to write and large in practice: a dbt project stops being one monolithic “build everything” job and becomes a library of selectable slices, each rendered into a DAG shaped for its purpose. The selector algebra is the vocabulary — union versus intersection, depth-limited graph walks, @ for the full rebuild set, method selectors for tags and configs and sources, and named selectors when the logic outgrows a one-liner. test_behavior and test_indirect_selection decide how thoroughly each slice checks itself, and source freshness decides whether the slice runs at all. Same models, same profile, different select — you’re not maintaining three projects; you’re pointing three DAGs at three views of one, and each one runs when there’s something new to do and stays out of the way when there isn’t. Which is the difference between a schedule and a schedule that respects your warehouse bill.

Next: Only Rebuild What Changed