XComs: How Tasks Pass Notes

When one task needs to tell the next one something small — a row count, a file path — it uses an XCom. Here's how they work, and the one way people misuse them.

Tasks in a DAG are isolated on purpose. Each one might run in a different process, a different container, a different machine, minutes or hours apart. So when extract finishes and load needs to know which file it just wrote, they can’t share a variable — there’s no shared memory to share it in. They pass a note instead, and the note is an XCom.

XCom is short for cross-communication, and that name is the whole spec: it’s how one task hands a small value across the gap to another. You’ve already been using them. Every TaskFlow return value is an XCom.

The TaskFlow way, and the classic way

In TaskFlow you never say the word “XCom.” A task returns, a downstream task takes the return as an argument, and the plumbing is invisible:

@task
def extract() -> str:
    return "/data/orders/2026-06-03.csv"

@task
def load(path: str):
    ...

load(extract())

extract pushes its return value to the XCom store; load pulls it back and binds it to path. You wrote a function call; Airflow ran a distributed handoff.

Sometimes you need the mechanism directly — usually inside a classic operator, or when a task pushes a value that no single return statement covers. Then you talk to the task instance, ti, which Airflow passes into your callable:

def extract(ti):
    ti.xcom_push(key="order_path", value="/data/orders/2026-06-03.csv")

def load(ti):
    path = ti.xcom_pull(task_ids="extract", key="order_path")
    ...

xcom_push writes a keyed value; xcom_pull reads it back by naming the task that produced it. This is exactly what TaskFlow does under the decorator — it’s the same push and pull, just written out. Reach for it when you’re outside TaskFlow’s return-value model; prefer the return value everywhere else, because it’s harder to get wrong.

One task, several notes

A task can push more than one XCom. The return value lands under return_value, and you can push extra keyed values alongside it — useful when a task produces a couple of independent facts that downstream tasks want separately rather than as one bundled dict:

def extract(ti):
    ti.xcom_push(key="order_path", value="/data/orders/2026-06-03.csv")
    ti.xcom_push(key="row_count", value=1204)

Now one downstream task can pull key="order_path" and another can pull key="row_count", each taking only the note it cares about. In TaskFlow, the clean way to get the same effect is multiple_outputs. Return a dict and set the flag, and Airflow splits the dict into one XCom per key instead of storing it as a single blob:

@task(multiple_outputs=True)
def extract() -> dict:
    return {"path": "/data/orders/2026-06-03.csv", "row_count": 1204}

info = extract()
load(info["path"])          # each key is its own XCom you can wire independently
report(info["row_count"])

Without the flag, info is one XCom holding the whole dict and info["path"] is a lazy lookup into it; with it, path and row_count are two separate XComs you can hand to two separate tasks. If your function’s return type is annotated as a dict, TaskFlow infers multiple_outputs=True for you, so you’ll sometimes get this behavior without asking — worth knowing when a return you expected to be one value shows up as several rows in the XComs view.

The xcom_pull surface, spelled out

That two-argument call hides a few defaults worth knowing, because the moment you have more than one producer or a non-return value, you’ll need them.

Start with key. When a TaskFlow task returns, Airflow stores that value under the key return_value — and xcom_pull defaults to exactly that key. So these two lines read the same XCom:

path = ti.xcom_pull(task_ids="extract")                    # key="return_value" implied
path = ti.xcom_pull(task_ids="extract", key="return_value")

That’s why pulling a TaskFlow return never mentions a key: the default is the return value. The moment you xcom_push(key="order_path", ...) by hand, you’ve opted out of the default and every pull has to name your key back. A common newcomer bug is pushing under a custom key and then pulling without one — you get None, because return_value was never written.

task_ids can be a list, and this is the feature people miss. Pass several task ids and you get several values back, in the order you asked for them:

def summarize(ti):
    counts = ti.xcom_pull(task_ids=["load_orders", "load_books", "load_authors"])
    # counts == [1204, 87, 63] — a list, positionally matched to the ids
    total = sum(counts)

One call, three notes collected. If any of those tasks was skipped or didn’t push, its slot comes back None rather than raising, so a downstream sum over a list that might contain None needs a guard. You can combine both arguments too — a list of task_ids with a single key pulls that same key from each named task.

Two more arguments come up in real pipelines. include_prior_dates=True widens the search beyond the current run: normally an XCom pull only sees values pushed by this logical run, so if you want yesterday’s row count to compare against today’s, you set this flag and Airflow will return the most recent matching XCom from an earlier run as well.

def compare_to_yesterday(ti):
    today = ti.xcom_pull(task_ids="count_rows")
    prior = ti.xcom_pull(
        task_ids="count_rows",
        include_prior_dates=True,   # reach back past this run
    )

And when you pull from a mapped task — one that fanned out into many parallel copies, which we’ll build in a moment — a plain pull gives you the whole list of every mapped copy’s value. If you want one specific copy, map_indexes selects it: map_indexes=0 returns just the first mapped instance’s value, and a list like map_indexes=[0, 2] returns those two. Leave it off and you get them all, in map order.

Where the note actually lives

Here’s the fact that dictates everything else: an XCom is stored in Airflow’s metadata database. When extract pushes a value, that value is serialized and written to a row in Postgres. When load pulls it, it reads that row back. The note travels through your database.

In Airflow 3.x, the task no longer touches that database directly. A worker running your task is sandboxed behind the Task Execution API — when your code calls xcom_push or xcom_pull, the Task SDK makes an API call to the api-server, and it does the database read or write on the task’s behalf. You don’t see this; your ti.xcom_pull looks identical to how it did in 2.x. But it’s why a task can run in a locked-down container with no database credentials at all and still pass notes: the broker in the middle holds the keys, not the worker. It’s also the seam a custom backend plugs into later.

Serialization matters here. By default Airflow serializes XCom values as JSON. That’s a deliberate, safe choice and it’s also a constraint: whatever you return has to be JSON-serializable — dicts, lists, strings, numbers, booleans, and None. A datetime, a set, a NumPy array, or a custom object won’t round-trip through the default backend without you converting it to something JSON understands first (an ISO string, a plain list). If you’ve used older Airflow, you may remember that pickling let you shove arbitrary Python objects through XCom. That path is gone — pickled XCom was deprecated and then removed, precisely because deserializing a pickle from your database is a code-execution footgun. JSON-only is the modern default, and you should treat it as the only default.

Because the value lands in a database column, size is bounded. The XCom value column has a finite width, and while the exact ceiling depends on your metadata database’s column type, the practical message is the same one the design is nudging you toward: an XCom is only ever meant to carry metadata — small, structured facts. A row count. A file path. A record id. A short status string. Things you’d be happy to see sitting in a database column, because that’s precisely where they’re sitting.

There’s a retry wrinkle that follows from the storage model too. An XCom row is keyed by DAG, task, run, and key — so when a task fails and retries, its second attempt’s push overwrites the first attempt’s value at the same key rather than appending a new one. That’s almost always what you want (the latest successful attempt’s note is the one downstream tasks should see), but it means you can’t treat XComs as an append-only log of every attempt. If you clear a task and re-run it, the same overwrite happens: the fresh run replaces the old note. One value per key per run is the invariant to keep in mind.

Classic operators can push too, whether you asked them to or not. Many operators return a value — a BashOperator returns the last line its command wrote to stdout, for instance — and Airflow stores that as the operator’s return_value XCom by default. Most of the time it’s harmless, but if an operator’s output is large or you simply don’t want it cluttering the store, set do_xcom_push=False on that operator and it won’t write one.

What the XComs view shows you

You don’t have to guess what got pushed. The React UI has a Browse → XComs page that lists every XCom row: which DAG, which task, which run, the key, and the serialized value. When a downstream task pulled a surprising value — or pulled None when you expected a path — this is the first place to look. You’ll often spot the bug immediately: the key you pushed under isn’t the key you pulled, or the task that should have produced the value was skipped and never wrote a row at all. Each task instance’s detail view also surfaces its own XComs, so you can trace a single run’s notes without scrolling the global list.

The view is also a gentle enforcement mechanism. If you open it and see a wall of enormous, truncated blobs, that’s your pipeline telling you it’s using XCom as a data pipe. Small, glanceable values are the healthy shape.

The anti-pattern, stated plainly

Do not pass data through XCom. Pass a handle to the data.

The tempting mistake is to have extract return the DataFrame it just built, or the contents of the CSV, and let load receive it as an argument — it reads so cleanly in TaskFlow that it feels free. It isn’t. That DataFrame gets serialized into a database row on push and deserialized on pull, so a fifty-megabyte extract becomes a fifty-megabyte write to your metadata DB, then a fifty-megabyte read — turning the database that keeps Airflow running into a data pipe it was never meant to be. Do it across enough runs and you’ll degrade the scheduler for every DAG on the instance, not just yours.

The fix is a one-word change in what you return. Don’t return the data — return where the data is:

from airflow.sdk import get_current_context, task

@task
def extract() -> dict:
    context = get_current_context()
    day = context["data_interval_start"].date().isoformat()
    path = f"/data/orders/{day}.csv"
    rows = fetch_orders_to(path)          # writes the file, returns a count
    return {"path": path, "row_count": rows}

@task
def load(info: dict):
    df = read_csv(info["path"])            # the next task reads it
    ...
    print(f"loaded {info['row_count']} rows")

extract writes the bytes to disk, object storage, or a warehouse — somewhere built to hold them — and passes only the path and a count through XCom. Both are tiny. load takes the path and reads the file itself. The big data moves through a real data store; the small note moves through XCom. Each carries what it’s for.

The row count riding along in that dict is a habit worth keeping. It costs nothing, it’s genuinely small, and it turns the XCom into a paper trail — you can look at any run in the UI and see how many rows moved, which is the first question you ask when a downstream number looks wrong.

If you want a rule of thumb for “how small is small,” use this: an XCom should be something you could read aloud. A path, an id, a count, a status, a short list of partition keys — those pass. A list of ten thousand ids, the parsed contents of a file, a base64-encoded anything — those fail, and the moment you catch yourself reaching for json.dumps to squeeze a structure into an XCom, stop and write it to storage instead. The test isn’t a byte count you have to memorize; it’s whether the value is a reference to work or the product of work. References travel through XCom. Products stay where they were made.

Feeding an XCom into a classic operator

TaskFlow binds return values for you, but half of Airflow is classic operators — BashOperator, SQLExecuteQueryOperator, and the rest — and those don’t take Python arguments. They take templated fields. So the way you hand an XCom to a classic operator is through a Jinja template that pulls it at render time:

from airflow.providers.standard.operators.bash import BashOperator

count_task = BashOperator(
    task_id="report_count",
    bash_command=(
        "echo 'loaded {{ ti.xcom_pull(task_ids=\"extract\")[\"row_count\"] }} "
        "orders from {{ ti.xcom_pull(task_ids=\"extract\")[\"path\"] }}'"
    ),
)

The {{ ... }} runs when the task starts, on the worker, with the same ti your Python callables get. ti.xcom_pull(task_ids="extract") returns the dict extract pushed, and the ["row_count"] subscript pulls the field out — all inside the template. Anything that’s a templated field on an operator can be fed this way, which is how you thread a value from a Python task into a Bash command, a SQL statement’s parameters, or a file path a downstream operator reads.

The same trick threads a value into a SQL step. Say a Python task figured out which partition to refresh, and a SQLExecuteQueryOperator needs to load exactly that path — the sql field is templated, so you drop the pull straight into it:

from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator

stage = SQLExecuteQueryOperator(
    task_id="stage_orders",
    conn_id="warehouse",
    sql=(
        "COPY INTO staging.orders "
        "FROM '{{ ti.xcom_pull(task_ids=\"extract\")[\"path\"] }}'"
    ),
)

When stage_orders starts, Airflow renders the template, the pull resolves the path extract wrote, and the warehouse sees a fully-formed COPY statement. The Python task decided what to load; the XCom carried the decision; the SQL operator executed it — and no data ever left its store to make that happen.

One thing the template does not do for you is create a dependency. When you write load(extract()) in TaskFlow, Airflow sees the wiring and orders the tasks. A raw xcom_pull inside a Jinja string is invisible to that graph-building step, so you have to state the edge yourself:

extract_task >> count_task

Forget it and count_task may render its template before extract ever ran, pulling None. The dependency arrow and the pull have to agree.

Fanning out over an XCom list

XComs also drive one of Airflow 3.x’s best features: dynamic task mapping. If a task returns a list, a downstream mapped task can .expand() over it and Airflow will create one parallel copy per element — at runtime, with the list length decided by the data, not hardcoded in the DAG.

Say the bookshop drops a variable number of regional order files each night, and you don’t know how many until you look:

@task
def list_order_files() -> list[str]:
    # returns however many files landed — could be 3, could be 30
    return glob("/data/orders/incoming/*.csv")

@task
def load_one(path: str) -> int:
    rows = load_csv_to_warehouse(path)
    return rows

counts = load_one.expand(path=list_order_files())

list_order_files returns a list through XCom. load_one.expand(path=...) tells Airflow: don’t call this once with the whole list — call it once per element, in parallel. Three files, three load_one instances; thirty files, thirty. Each mapped copy pushes its own return value, and counts is the collection of all of them — which is exactly the mapped XCom that map_indexes selects into, and which a plain xcom_pull returns whole.

The natural next step is to fan back in: a single downstream task that takes the whole counts collection and reduces it. Because counts is already an XCom reference to every mapped result, you just pass it as a normal argument and Airflow resolves it into the full list:

@task
def report_total(per_file_counts: list[int]):
    total = sum(per_file_counts)
    print(f"loaded {total} orders across {len(per_file_counts)} files")

report_total(counts)

report_total runs once, after every load_one copy has finished, and receives [1204, 87, 63, ...] — one integer per mapped instance, in map order. That’s the classic map/reduce shape expressed entirely through XCom: a list fans a task out, and the collected return values fan it back in.

This is the payoff of the “small notes” discipline arriving in a new form. The mapping is driven by a list of paths — a handful of short strings — not by the file contents. The data stays on disk; the XCom carries the manifest, and the reduce carries only the counts.

When the handoff has to be bigger

Occasionally you genuinely need to pass something that won’t fit the “small note” mold, and rewriting to a shared path isn’t clean. For that, Airflow lets you configure a custom XCom backend — you swap the storage behind the XCom API so that values are written to object storage like S3 or GCS instead of the metadata DB, while your DAG code still just returns and receives. The tasks keep passing notes; the notes just get stored somewhere with room.

The mechanism is a subclass. A backend is a class that inherits from BaseXCom and overrides two methods — serialize_value, called on push, and deserialize_value, called on pull:

from airflow.sdk import BaseXCom

class S3XComBackend(BaseXCom):
    @staticmethod
    def serialize_value(value, **kwargs):
        # write the real payload to S3, keep only a URI in the metadata DB
        key = upload_to_s3(value)                     # returns "s3://bucket/xcom/…"
        return BaseXCom.serialize_value(f"s3://{key}")

    @staticmethod
    def deserialize_value(result):
        uri = BaseXCom.deserialize_value(result)      # read the URI back
        return download_from_s3(uri)                  # fetch the real payload

On push, serialize_value sends the bulky value to S3 and stores only a compact URI in the database. On pull, deserialize_value reads the URI and fetches the object back. The task code — return df, df = upstream() — never changes; the storage underneath it did. You wire the backend in globally, not per-DAG, with an environment variable:

AIRFLOW__CORE__XCOM_BACKEND=my_plugins.xcom.S3XComBackend

Because XCom in 3.x flows through the Task Execution API, the backend runs where that broker runs, which keeps object-store credentials out of the task sandbox — the same separation that already lets a task push notes without a database password. This is deliberately just the sketch; the real build (staging, cleanup of orphaned objects, previewing a lightweight value in the UI so the XComs page doesn’t try to download a gigabyte) has enough corners that it gets its own chapter in the In Practice series: Custom XCom Backends. Treat the custom backend as the escape hatch, not the default — most pipelines never need it, and if you find yourself wanting it constantly, that’s usually a sign a few tasks should be passing paths instead of payloads.

Final thoughts

The mental model that keeps you out of trouble: XComs move pointers, not payloads. A task’s job is to put its output somewhere durable and tell the next task where to look — the same discipline you’d use handing work between two services that don’t share a disk. The xcom_pull arguments, the templated {{ ti.xcom_pull(...) }}, the .expand() fan-out — they’re all just ways of moving small, structured notes around the graph, and they stay cheap precisely because the notes stay small. Get that instinct early and your DAGs scale from a bookshop’s daily orders to something a thousand times larger without touching the pattern. Ignore it and the ceiling arrives fast, wearing the disguise of a slow, mysteriously unhappy scheduler.

Next: Schedules, Catchup, and Backfills Without the Fear

Comments