Templating, Macros, and Rendered Fields

You met {{ ds }} back in the operators post, tucked inside a bash_command, and I promised templating was a feature of its own. This is that post. The one-line version of the whole thing: Airflow templating is not global string interpolation. It doesn’t reach into every string in your DAG file and swap in dates. It runs Jinja on a small, declared set of fields on each operator — the fields the operator’s author marked as templatable — and nowhere else. Understand which fields those are and where the values come from, and templating stops being magic. Miss that one rule and you’ll write {{ ds }} in the wrong place, watch it come out as the literal five characters {{ ds }}, and lose an afternoon.

The rule that explains everything: `template_fields`

Every operator class has a class attribute called template_fields — a tuple of the argument names that get rendered as Jinja before the task runs. BashOperator declares template_fields = ("bash_command", "env", "cwd"). SQLExecuteQueryOperator declares template_fields = ("sql", "conn_id", "parameters", ...). When the scheduler is about to run a task, it walks that tuple, takes the string you passed for each named field, renders it against the run’s context, and hands the rendered string to the operator’s execution logic. A value you passed to any argument not in template_fields is passed through untouched — {{ ds }} in there is just text.

So this works, because bash_command is templatable:

from airflow.providers.standard.operators.bash import BashOperator

fetch = BashOperator(
    task_id="fetch_orders",
    bash_command="curl -sf https://bookshop.internal/orders/{{ ds }}.csv "
                 "-o /data/orders/{{ ds }}.csv",
)

Airflow renders {{ ds }} to 2026-07-04 and the shell sees a real date. Good.

Now the trap. Here is the single most common newcomer mistake in Airflow 3, and it’s worth staring at:

from airflow.sdk import task

@task
def build_path():
    # WRONG. This returns the literal string "{{ ds }}".
    return "/data/orders/{{ ds }}.csv"

A plain @task return value is not a template field. Nothing renders it. The function returns exactly what you wrote — the eight characters of {{ ds }} sitting in the middle of your path — and every downstream task that reads that XCom gets a broken filename. Jinja only fires inside template_fields, and a Python function’s return value is not one of them.

The fix is to stop thinking of Jinja as the way to get the date inside a @task. Inside your own Python, you read the run context directly:

from airflow.sdk import task

@task
def build_path(data_interval_start=None):
    # RIGHT. data_interval_start is a real datetime, injected by the Task SDK.
    return f"/data/orders/{data_interval_start:%Y-%m-%d}.csv"

The Task SDK injects context values into your callable when you declare them as parameters — data_interval_start, data_interval_end, logical_date, ds, ti, params, dag_run, and the rest. No Jinja involved, because there’s no template field involved. You’re in Python; use Python. Jinja is for the operator arguments you can’t reach with code — the bash_command string, the .sql file, the env dict — the places where the value has to be a rendered string before the operator ever runs your logic.

That division is the whole mental model. Jinja renders in operator template_fields. Inside a @task body, read the context. If you can hold those two sentences in your head, the rest of this post is detail.

But there’s a wrinkle in the middle that trips people who take the rule too literally, so let’s settle it now: a @task’s arguments do render. The decorator builds an operator whose template_fields are ("op_args", "op_kwargs") — that is, the values you pass in to the task. So this works exactly as you’d hope:

@task
def load(batch_size: str, day: str):
    print(batch_size, day)      # '500'  '2026-07-04'

load(batch_size="{{ var.value.bookshop_batch_size }}", day="{{ ds }}")

The line to hold is therefore sharper than “Jinja doesn’t work in TaskFlow.” It’s about where the string lives when Airflow looks at it. A template in an argument is rendered, because the argument is a field on the operator and Airflow renders it before your function is ever called. A template in a string you build inside the function body is not, because by then Airflow has already finished rendering and handed control to your Python — which is exactly why f"/data/{{ ds }}.csv" inside a task gives you literal braces. Arguments in: rendered. Anything you construct in the body: yours to compute from the context.

One caveat worth knowing: what arrives is always a string. batch_size above is '500', not 500. If you need a real int, cast it — or set render_template_as_native_obj=True on the DAG, which we come back to later in this chapter.

Where the values come from: the template context

When Airflow renders a template field, it does it against the template context — a dictionary of everything known about this particular run. These are the names you can put inside {{ }}. The ones tied to the run’s schedule are the ones you’ll use daily:

{{ ds }} — the logical date as YYYY-MM-DD, e.g. 2026-07-04. The workhorse.
{{ ds_nodash }} — the same date as 20260704. Handy for filenames and partition keys that dislike dashes.
{{ ts }} — a full ISO timestamp, 2026-07-04T00:00:00+00:00.
{{ ts_nodash }} — 20260704T000000, for when even the colons are inconvenient.
{{ logical_date }} — the run’s logical date as a pendulum datetime. This replaced execution_date, which is gone in Airflow 3.
{{ data_interval_start }} and {{ data_interval_end }} — the datetimes bounding the period this run covers. For an @daily DAG, start is midnight of the day and end is midnight of the next day. Prefer these over ds when you mean “the window of data I’m processing,” because they say exactly what they name. ds is a convenient string; the interval is the truth about what this run is responsible for.
{{ dag_run }} — the DagRun object, with .run_id, .run_type (scheduled, manual, backfill), and .conf (the payload from a manually triggered or API-triggered run).
{{ ti }} — the TaskInstance, mostly used for {{ ti.xcom_pull(task_ids='...') }} to read another task’s output right inside a template.
{{ params }} — the DAG’s params, which we’ll come back to. Access as {{ params.reprocess }}.
{{ var.value.my_key }} — an Airflow Variable, rendered lazily so it’s only fetched if the template actually uses it. {{ var.json.my_key }} parses a JSON-valued variable so you can do {{ var.json.settings.batch_size }}.
{{ conn.my_conn_id }} — a Connection object, so {{ conn.bookshop_duckdb.host }} pulls a field out of a connection without hardcoding it.
{{ macros }} — a namespace of helper functions, covered below.
{{ prev_data_interval_start_success }} — the data_interval_start of the last successful run of this DAG. Priceless for incremental loads: “give me everything since the last time I actually succeeded,” which correctly skips the gap left by a failed run instead of silently dropping a day.

A worked line that uses several at once — a shell command that syncs one day’s partition and stamps the run id into the log:

BashOperator(
    task_id="archive_day",
    bash_command=(
        "aws s3 sync /data/orders/{{ ds }}/ "
        "s3://bookshop-archive/orders/dt={{ ds_nodash }}/ "
        "--metadata run={{ dag_run.run_id }}"
    ),
)

`ds` versus the interval, made concrete

The reason to prefer data_interval_start/_end over ds isn’t pedantry — it’s what happens during a backfill. Say it’s July and you realize June’s rollups were computed with a bug. You backfill the whole month. Airflow now creates one run per day of June, and each run gets its own context: the run “for” June 3rd has data_interval_start = 2026-06-03 and data_interval_end = 2026-06-04, regardless of the fact that it’s physically executing in July. Your templated WHERE order_date >= '{{ data_interval_start | ds }}' AND order_date < '{{ data_interval_end | ds }}' therefore processes exactly June 3rd’s rows in that run, exactly June 4th’s in the next, and so on across the month. The interval is the contract: each run owns one window, the windows tile the month with no gaps and no overlaps, and the same DAG file backfills correctly without you editing a thing. Write it against ds alone and the query is ambiguous about which window it means; write it against the interval and it’s unambiguous, which is the whole reason backfills are trustworthy.

{{ prev_data_interval_start_success }} extends the same idea to incremental loads. Instead of “the fixed window for this run,” it gives you “everything since the last run that actually succeeded” — so a load that failed Tuesday and reran Wednesday picks up Tuesday’s data too, rather than leaving a silent hole. The distinction between “scheduled window” and “since last success” is exactly the kind of thing you want the template to state out loud.

`macros`: the small standard library inside a template

{{ macros }} is a namespace of functions you can call from Jinja for the date arithmetic that comes up constantly. The one you’ll reach for most:

{{ macros.ds_add(ds, 7) }}     -- ds shifted forward 7 days
{{ macros.ds_add(ds, -1) }}    -- yesterday, as a string
{{ macros.ds_format(ds, "%Y-%m-%d", "%Y/%m/%d") }}   -- reformat a date string

macros also exposes datetime, timedelta, and dateutil so you can do real date math inline: {{ (data_interval_start - macros.timedelta(days=1)).strftime("%Y%m%d") }}. The temptation is to cram logic in here; resist it past a certain point. A one-liner shift like ds_add(ds, -1) is clear at a glance. A three-clause conditional buried in Jinja is a debugging nightmare, because Jinja has no debugger and no stack trace — it just renders wrong and hands you a bad string. When the logic grows, move it into a @task and read the context there.

Templating whole files: `.sql` and `.sh`

Inlining a big SQL query as a Python string is miserable — no syntax highlighting, no formatting, no reviewer can read the diff. Airflow’s answer is that a template field can name a file, and Airflow will load and render the file’s contents instead of treating the string as the template itself. Two settings on the DAG control this:

template_searchpath — a directory (or list of them) Airflow will search for template files, in addition to the DAG’s own folder.
template_ext — the file extensions that trigger file-loading. On SQLExecuteQueryOperator this includes .sql; on BashOperator, .sh. When the string you pass to a template field ends in one of these extensions, Airflow reads that file and renders its contents.

So when you write sql="sql/daily_rollup.sql", the operator sees a string ending in .sql, finds daily_rollup.sql on the search path, loads it, and renders any Jinja inside the file against the run context. The file is the template.

Here’s the worked bookshop example. Project layout:

dags/
  bookshop_rollup.py
  sql/
    daily_rollup.sql

The SQL file, with Jinja in it:

-- dags/sql/daily_rollup.sql
INSERT INTO analytics.daily_book_sales
SELECT
    o.order_date,
    o.book_id,
    b.title,
    SUM(o.quantity)        AS units_sold,
    SUM(o.quantity * o.unit_price) AS revenue
FROM raw.orders o
JOIN raw.books b USING (book_id)
WHERE o.order_date >= '{{ data_interval_start | ds }}'
  AND o.order_date <  '{{ data_interval_end | ds }}'
GROUP BY 1, 2, 3;

The | ds is a built-in Jinja filter that formats a datetime as YYYY-MM-DD — so the interval datetimes come out as clean date literals. And notice the >= start AND < end window: it’s half-open, so a day’s rows land in exactly one run with no double-counting at the boundary. That’s the idempotency habit from the operators post, expressed in SQL.

The DAG that wires it up:

from airflow.sdk import DAG
from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator

with DAG(
    dag_id="bookshop_rollup",
    schedule="@daily",
    catchup=False,
    template_searchpath="/usr/local/airflow/dags",   # so "sql/..." resolves
) as dag:
    rollup = SQLExecuteQueryOperator(
        task_id="daily_rollup",
        conn_id="bookshop_duckdb",
        sql="sql/daily_rollup.sql",
    )

The path in sql= is relative to the search path (and Airflow also looks in the DAG file’s own directory, which is why simple co-located files often work with no template_searchpath at all). Now the query lives in a real .sql file your team can lint and review, the dates are injected per run, and the DAG file stays about orchestration instead of SQL.

The identical mechanism works for shell scripts: point bash_command at a .sh file on the search path and Airflow renders the script’s contents. Same rule, different extension.

Bridging a `@task` output into a template

Remember the broken example — the @task that returned "{{ ds }}.csv" and got the literal braces? Now that you know where Jinja does and doesn’t fire, here’s the clean pattern for the real goal behind that mistake: compute a value in Python, then feed it into an operator’s template field. You don’t template inside the @task; you return a plain value, and you pull it into the operator with xcom_pull from a template field, where Jinja does run.

from airflow.sdk import DAG, task
from airflow.providers.standard.operators.bash import BashOperator

with DAG(dag_id="bookshop_ingest", schedule="@daily", catchup=False) as dag:

    @task
    def source_path(data_interval_start=None):
        # plain Python, plain return — no Jinja here
        return f"/data/orders/{data_interval_start:%Y-%m-%d}.csv"

    path = source_path()

    validate = BashOperator(
        task_id="validate",
        # Jinja DOES run here, because bash_command is a template field
        bash_command="test -s {{ ti.xcom_pull(task_ids='source_path') }}",
    )

    path >> validate

ti.xcom_pull inside the bash_command reads the value the @task pushed to XCom, and because it’s inside a template field it actually renders. The Python computed the path with real datetime formatting; the operator consumed it through Jinja. That’s the division of labour the whole post is about, shown end to end: context-reading in Python, string-rendering in the operator.

Rendering happens at run time, not parse time

One timing fact prevents a class of confusion. Templates render when a task instance runs, not when the DAG file is parsed. At parse time there is no run, no logical date, no ds — the scheduler is just building the graph. This is the same reason you can’t loop over runtime data at the top of a DAG file (the dynamic-mapping post’s whole premise). So templating can change what a task does — which date it queries, which file it reads — but it cannot change the shape of the DAG: you can’t template a task_id, add or remove tasks, or redraw dependencies with Jinja, because all of that is decided at parse time before any {{ }} is ever evaluated. Templates fill in values; they don’t build graphs.

Nested fields render too

A template field doesn’t have to be a bare string. Airflow renders recursively into lists and dicts, so a templatable env dict on BashOperator renders each value:

BashOperator(
    task_id="load",
    bash_command="./load.sh",
    env={
        "RUN_DATE": "{{ ds }}",
        "PARTITION": "{{ ds_nodash }}",
        "SCHEMA": "{{ params.target_schema }}",
    },
)

Each value in that dict is rendered; the keys are not. The same applies to a list passed to a templatable argument. You don’t have to flatten structured arguments into one big string to get templating — as long as the top-level argument is in template_fields, Airflow reaches down into the structure for you.

`params`: inputs a human sets at trigger time

Context values like ds describe the run. Sometimes you want an input the operator sets when they trigger the DAG — “reprocess the last 30 days,” “run against staging.” That’s params. You declare them on the DAG with defaults, and they surface as a form in the UI’s Trigger dialog and as {{ params.* }} in templates.

from airflow.sdk import DAG, Param

with DAG(
    dag_id="bookshop_rollup",
    schedule="@daily",
    catchup=False,
    template_searchpath="/usr/local/airflow/dags",
    params={
        "target_schema": Param("analytics", type="string"),
        "full_refresh": Param(False, type="boolean"),
    },
) as dag:
    ...

Inside the SQL file you can now branch on them: INSERT INTO {{ params.target_schema }}.daily_book_sales ..., or gate a TRUNCATE behind {% if params.full_refresh %}. Param enforces a type and can carry a JSON-schema validation, so a bad trigger payload is rejected before the run starts rather than blowing up mid-task. A scheduled run uses the defaults; a manual run lets a human override them from the form.

Debugging: the Rendered Template view

Here is the tool that turns templating from guesswork into a two-click check, and it’s the thing to teach yourself to open first when a templated task misbehaves. In the React UI, open a task instance and choose the Rendered Template tab. It shows you, field by field, the exact string Airflow produced after rendering — the real bash_command, the fully-substituted .sql with dates filled in, the resolved env. Not what you wrote; what actually ran.

This collapses an entire category of bug. A SQL task loaded the wrong day? Don’t reach for the database logs yet — open Rendered Template and read the WHERE clause. If it says order_date >= '2026-07-03' when you expected the 4th, the bug is in your template or your understanding of the interval, and you’ve found it in ten seconds without touching the warehouse. If it says the right date and the data’s still wrong, now you know the problem is downstream. The Rendered Template view answers the question “did Jinja do what I think?” definitively, every time. It’s available even for a task instance that failed, so a rendering mistake is visible without a successful run.

Knowing a field is templatable

When you’re unsure whether an argument renders, don’t guess — check the operator’s template_fields. Every operator advertises it, and you can read it straight off the class:

>>> from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator
>>> SQLExecuteQueryOperator.template_fields
('sql', 'parameters', 'conn_id', 'database', 'hook_params')

That tuple is the definitive answer to “can I put {{ ds }} here?” for that operator. It also explains asymmetries that otherwise look arbitrary: sql renders because it’s in the tuple, but a helper argument you invented on a subclass won’t render unless you add it to template_fields. If you ever write a custom operator (a later post in the in Practice series), extending template_fields is exactly how you make one of your own arguments Jinja-aware — it’s the same mechanism the built-in operators use, not a privileged one.

`render_template_as_native_obj`: when a string won’t do

By default Jinja renders to a string. {{ params.batch_size }} where batch_size is the integer 500 comes out as the string "500". Usually fine — a shell command wants a string, SQL wants a string. But if a templated field needs to pass a real Python int, list, or dict to the operator, a string breaks it.

Set render_template_as_native_obj=True on the DAG and Airflow uses Jinja’s native environment: a template that resolves to a single expression yields the underlying Python object, not its string form.

with DAG(
    dag_id="bookshop_load",
    schedule="@daily",
    catchup=False,
    render_template_as_native_obj=True,
    params={"book_ids": Param([1, 2, 3], type="array")},
) as dag:
    ...

Now {{ params.book_ids }} in a template field arrives as an actual list, and {{ var.json.settings.batch_size }} arrives as an int you can compare and do arithmetic on. It’s a DAG-wide switch, so turn it on when you’re deliberately passing structured data through templates and leave it off otherwise — most DAGs never need it, and it slightly changes how borderline strings coerce.

`user_defined_macros` and `user_defined_filters`

The built-in macros namespace covers dates. For your own project conventions, the DAG takes two hooks:

user_defined_macros — a dict of callables (or values) added to the template context. user_defined_macros={"s3_bucket": "bookshop-archive", "fmt_dt": my_fn} lets any template in that DAG write {{ s3_bucket }} or {{ fmt_dt(ds) }}.
user_defined_filters — a dict of functions usable with Jinja’s pipe syntax. user_defined_filters={"to_partition": lambda d: d.replace("-", "/")} lets you write {{ ds | to_partition }} to get 2026/07/04.

def to_partition(ds: str) -> str:
    return ds.replace("-", "/")

with DAG(
    dag_id="bookshop_archive",
    schedule="@daily",
    catchup=False,
    user_defined_macros={"archive_bucket": "bookshop-archive"},
    user_defined_filters={"to_partition": to_partition},
) as dag:
    archive = BashOperator(
        task_id="archive_day",
        bash_command=(
            "aws s3 sync /data/orders/{{ ds }}/ "
            "s3://{{ archive_bucket }}/orders/{{ ds | to_partition }}/"
        ),
    )

That renders to s3://bookshop-archive/orders/2026/07/04/. The value of this is a single source of truth for a convention — change the bucket or the partition scheme in one place and every template follows. The value ends the moment you start hiding business logic in there. A macro named to_partition that reformats a date is a convenience; a macro that decides which schema to write to based on the day of the week is a program smuggled into a string, invisible to code search, untestable, and undebuggable. If a template function is doing more than formatting, it belongs in a @task where it has a name, a test, and a stack trace.

Final thoughts

Templating earns its keep by letting one static DAG file describe every run — today’s, yesterday’s, the backfill from March — without a single edit, because the dates and inputs arrive at render time. But it’s a narrow tool with sharp edges, and the whole of it fits in a few rules. Jinja renders only in an operator’s template_fields, never in a plain @task return — inside your Python, read data_interval_start from the context instead. File extensions and template_searchpath let you keep real SQL and shell in real files. The context gives you dates, params, variables, and connections; macros and your own filters format them. And when any of it looks wrong, the Rendered Template view shows you exactly what ran before you go blaming the database. Keep templates about runtime parameters made explicit — dates, paths, ids — and keep logic in code, and templating stays the quiet convenience it’s meant to be instead of a second, worse programming language living inside your strings.

Next: Connections, Variables, and Secrets Backends

The rule that explains everything: template_fields

Where the values come from: the template context

ds versus the interval, made concrete

macros: the small standard library inside a template

Templating whole files: .sql and .sh

Bridging a @task output into a template