Kubernetes and Multi-Executor Patterns

Per-task pods, pod overrides, templates, CeleryKubernetes tradeoffs, and the triggerer role that cuts across executors.

The executors chapter gave you the map: Local for one host, Celery for warm horizontal scale, Kubernetes for per-task isolation, a list of them when you want both. It ended by pointing here for the parts an overview can’t fit — the pod lifecycle, the base template, the merge rules that decide what your task’s pod actually looks like, and the one distinction that trips up almost everyone the first time: KubernetesExecutor and KubernetesPodOperator are not the same thing, and you can run both at once.

This is the last chapter of the series, so we’re going to go all the way down. By the end you’ll be able to reason about a pod from the moment the scheduler decides a task is ready to the moment Kubernetes garbage-collects it, shape that pod per-task without touching any other, run cheap steady work on Celery and heavy isolated work on Kubernetes in the same DAG, and autoscale the whole thing down to nothing overnight. Then we’ll wrap the series.

What actually happens when a task runs under KubernetesExecutor

Under Celery, a ready task is a message on a broker that a warm worker pulls. Under KubernetesExecutor there is no broker and no warm worker. When the scheduler marks a bookshop task instance ready, the executor calls the Kubernetes API and asks it to create a pod — one pod, for that one task instance. The pod’s single container runs airflow tasks run for exactly that DAG, task, and run, streams its logs, exits, and the pod is torn down. Nothing about it survives to the next task.

That has consequences worth making explicit, because they explain every operational quirk of this executor:

  • The scheduler talks to the Kubernetes API, not to workers. There’s a component inside the scheduler process — historically called the Kubernetes job watcher — that opens a watch stream on the API server and receives pod status events (Pending → Running → Succeeded/Failed). That’s how Airflow learns a task finished: not from a worker reporting back over a result backend, but from Kubernetes telling it the pod’s phase changed. If the watch stream drops, the executor reconnects and resumes from the last resource version it saw, so it doesn’t lose track of pods across a blip.
  • Pod creation is rate-limited on purpose. The scheduler doesn’t fire a thousand create pod calls at the API server the instant a big DAG unblocks. It creates them in batches, governed by worker_pods_creation_batch_size (default 1 — yes, one per scheduler loop unless you raise it). On a fan-out of hundreds of mapped tasks this is the difference between a smooth ramp and hammering the API server, and it’s the first knob to raise when Kubernetes tasks are slow to start even though the cluster has capacity.
  • Orphaned pods get adopted. If the scheduler restarts while pods are running, the new scheduler doesn’t abandon them. On startup it lists pods carrying its Airflow labels and adopts the running ones, resuming the watch rather than starting the tasks over — the Kubernetes analogue of Celery’s task adoption. This is why a rolling scheduler restart mid-run doesn’t kill in-flight Kubernetes tasks.

The tradeoff is the one the overview named, and now you can see exactly where it comes from: every task pays pod-scheduling and image-pull latency at the front. The kube-scheduler has to place the pod, the node may have to pull the worker image if it’s cold, the container has to start, and only then does your task code begin. That’s seconds — sometimes tens of seconds on a cold node pulling a fat image — added to every task instance, invisible on a 20-minute transform and brutal on a thousand two-second sensors. The flip side is that when the task ends, the pod ends, and with a cluster autoscaler the node eventually ends too. You scale to zero for free; you just pay a cold start to scale back up.

The [kubernetes_executor] config that matters

The executor’s behavior lives under [kubernetes_executor] (env vars are AIRFLOW__KUBERNETES_EXECUTOR__*). The handful you’ll actually touch:

[kubernetes_executor]
# Where task pods are created
namespace = airflow

# The base pod spec every task pod is built from (more below)
pod_template_file = /opt/airflow/pod_templates/pod_template.yaml

# The image task pods run, if not set in the template
worker_container_repository = my-registry/airflow
worker_container_tag = 3.3.0

# How many pods the scheduler creates per loop — raise for high fan-out
worker_pods_creation_batch_size = 16

# Clean up finished pods. Keep failed ones for debugging.
delete_worker_pods = True
delete_worker_pods_on_failure = False

# Watch pods across many namespaces (multi-tenant clusters)
multi_namespace_mode = False

Two of these decide how much you can debug and how much your cluster fills with corpses. delete_worker_pods = True (the default) tears down every pod when its task finishes — tidy, but a failed task’s pod vanishes and with it the easy kubectl logs post-mortem. The common production compromise is exactly what’s shown: delete successful pods, keep failed ones (delete_worker_pods_on_failure = False) so you can inspect the container that broke. Just remember failed pods then accumulate and someone has to sweep them, or they’ll pin nodes and quietly cost money.

namespace is where task pods land, and it doesn’t have to be the namespace the scheduler runs in. On a shared cluster you’ll often give Airflow its own namespace with a resource quota, so a runaway DAG can’t starve everything else on the cluster. multi_namespace_mode plus a list of namespaces lets one deployment scatter task pods across several — useful for genuine multi-tenancy, at the cost of broader RBAC (the scheduler’s service account now needs pod permissions in every one of those namespaces).

The base pod template, and how pod_override merges into it

Every task pod starts from a base pod template — a plain Kubernetes Pod manifest at pod_template_file. This is where you put everything common to all tasks so no DAG has to repeat it: the image, the service account, the security context, the log-sidecar arrangement, default resource requests, the standard volumes. Think of it as the pod your tasks run in by default.

# pod_template.yaml — the baseline every bookshop task pod inherits
apiVersion: v1
kind: Pod
metadata:
  name: airflow-worker
spec:
  serviceAccountName: airflow-worker
  restartPolicy: Never
  securityContext:
    runAsUser: 50000
    fsGroup: 0
  containers:
    - name: base            # <-- this name is load-bearing; see below
      image: my-registry/airflow:3.3.0
      resources:
        requests:
          cpu: "250m"
          memory: "512Mi"
        limits:
          cpu: "1"
          memory: "1Gi"
      volumeMounts:
        - name: dags
          mountPath: /opt/airflow/dags
  volumes:
    - name: dags
      persistentVolumeClaim:
        claimName: airflow-dags

Now the per-task part. A task’s executor_config["pod_override"] is a full V1Pod that Airflow strategically merges over this base. The rule that catches everyone: the container you want to modify must be named base. Airflow matches containers by name when merging, so a pod_override container named anything else is added as a second container rather than overriding the first — and your resource bump silently does nothing while the task keeps running with the template’s defaults. Name it base, override only the fields you care about, and leave the rest to inherit.

Here’s a bookshop task that needs a genuinely big pod — a nightly rebuild of the search index that loads the whole catalog into memory and wants to land on the memory-optimized node pool, tolerating that pool’s taint:

from airflow.sdk import dag, task
from kubernetes.client import models as k8s

BIG_POD = k8s.V1Pod(
    spec=k8s.V1PodSpec(
        # Land only on the memory-optimized pool...
        node_selector={"workload": "mem-optimized"},
        # ...which is tainted so nothing else schedules there.
        tolerations=[
            k8s.V1Toleration(
                key="dedicated",
                operator="Equal",
                value="mem-optimized",
                effect="NoSchedule",
            )
        ],
        # Spread these pods across zones rather than stacking them.
        affinity=k8s.V1Affinity(
            pod_anti_affinity=k8s.V1PodAntiAffinity(
                preferred_during_scheduling_ignored_during_execution=[
                    k8s.V1WeightedPodAffinityTerm(
                        weight=100,
                        pod_affinity_term=k8s.V1PodAffinityTerm(
                            topology_key="topology.kubernetes.io/zone",
                            label_selector=k8s.V1LabelSelector(
                                match_labels={"task": "rebuild-search-index"}
                            ),
                        ),
                    )
                ]
            )
        ),
        containers=[
            k8s.V1Container(
                name="base",  # MUST be "base" to override, not add
                resources=k8s.V1ResourceRequirements(
                    requests={"cpu": "2", "memory": "12Gi"},
                    limits={"cpu": "4", "memory": "16Gi"},
                ),
                volume_mounts=[
                    k8s.V1VolumeMount(name="scratch", mount_path="/scratch")
                ],
            )
        ],
        volumes=[
            k8s.V1Volume(
                name="scratch",
                empty_dir=k8s.V1EmptyDirVolumeSource(
                    size_limit="20Gi", medium=""
                ),
            )
        ],
    )
)


@dag(schedule="@daily", catchup=False)
def bookshop_search():
    @task  # inherits the template: 250m CPU / 512Mi, default node pool
    def extract_catalog_delta():
        ...

    @task(executor_config={"pod_override": BIG_POD})
    def rebuild_search_index():
        # runs in a 16Gi pod on the mem-optimized pool
        ...

    @task  # back to a normal small pod
    def notify_search_ready():
        ...

    extract_catalog_delta() >> rebuild_search_index() >> notify_search_ready()


bookshop_search()

The two ordinary tasks around it get the template’s modest 512Mi pod on the default node pool; only rebuild_search_index gets the 16Gi behemoth, the node selector, the toleration, and the scratch volume. Nothing about the big task leaks onto the small ones, and nothing about the template had to change. That is the entire point of the executor: the base template is the shared floor, and pod_override is the per-task delta.

A few things worth internalizing about the merge. It’s a strategic merge, so lists like volumes, volumeMounts, env, and tolerations are combined by key rather than wholesale-replaced — your override adds the scratch volume without clobbering the dags volume from the template. Fields you don’t mention are inherited untouched. And because pod_override is real Python building real V1Pod objects, you can factor the common overrides into a helper and stamp them across many tasks — a gpu_pod(memory="8Gi") function beats copy-pasting fifty lines of k8s.* per task. Just keep the object at module scope or inside a factory, not built at DAG-parse time in a way that hits the network, or you’ll pay for it on every scheduler parse.

Multiple executors: cheap on Celery, heavy on Kubernetes, one DAG

Everything above assumes the whole deployment runs KubernetesExecutor. But most real bookshop workloads are lopsided: a hundred small, similar, latency-sensitive tasks — poll the orders API, run a dbt model, send a Slack ping — and a handful of monsters like the index rebuild. Run all of it on Kubernetes and the hundred small tasks each eat a pod cold-start they can’t afford. Run all of it on Celery and the monsters have to fit inside a shared worker’s RAM.

The 2.x answer was CeleryKubernetesExecutor, a hybrid that inspected a magic kubernetes queue name and shunted those tasks to pods. It worked and it was a wart. Airflow 3 supersedes it entirely with first-class multiple executors: you configure a list, and each task names the one it wants.

[core]
# First entry is the default. Aliases keep the per-task strings tidy.
executor = CeleryExecutor,KubernetesExecutor

You can alias them so DAG code reads cleanly and you’re free to swap the implementation behind a name later:

executor = airflow.providers.celery.executors.celery_executor.CeleryExecutor:celery,airflow.providers.cncf.kubernetes.executors.kubernetes_executor.KubernetesExecutor:k8s

Now the same bookshop nightly DAG runs its steady tasks warm and its heavy task isolated:

from airflow.sdk import dag, task
from kubernetes.client import models as k8s


@dag(schedule="@daily", catchup=False)
def bookshop_nightly():
    @task  # default → CeleryExecutor: warm worker, ~zero startup
    def poll_orders_api():
        ...

    @task(executor="celery")  # explicit, same as default
    def run_dbt_marts():
        ...

    @task(  # its own 16Gi pod, its own limits, its own blast radius
        executor="k8s",
        executor_config={
            "pod_override": k8s.V1Pod(
                spec=k8s.V1PodSpec(
                    containers=[
                        k8s.V1Container(
                            name="base",
                            resources=k8s.V1ResourceRequirements(
                                requests={"memory": "12Gi"},
                                limits={"memory": "16Gi"},
                            ),
                        )
                    ]
                )
            )
        },
    )
    def rebuild_search_index():
        ...

    poll_orders_api() >> run_dbt_marts() >> rebuild_search_index()


bookshop_nightly()

Notice executor_config only means something to the executor that understands it — pod_override is honored by k8s and ignored by celery, so the two coexist without conflict. The cost of this feature is real and worth stating plainly: each executor in the list needs its whole substrate present and healthy. Listing CeleryExecutor,KubernetesExecutor means you now operate a broker and a result backend and a worker fleet and a Kubernetes cluster with the right RBAC — the union of both operational surfaces, not a clever way to avoid either. The feature is elegant; the ops bill is additive. Reach for it when your workload genuinely has two shapes, not because the config line is easy to type.

KEDA: autoscaling Celery down to zero

If you are running Celery (on Kubernetes, via the Helm chart), the elasticity gap versus Kubernetes-per-task is closed by KEDA. The executors chapter showed the shape; here’s the operational depth. KEDA runs a controller that scales the worker Deployment against a metric, and for Airflow the right metric is queue depth measured straight from the metadata DB — count the task instances that are queued or running and divide by worker concurrency:

workers:
  keda:
    enabled: true
    minReplicaCount: 0        # scale-to-zero when idle
    maxReplicaCount: 12
    pollingInterval: 5        # check the metric every 5s
    cooldownPeriod: 300       # wait 5 min of idle before scaling in
    query: >-
      SELECT ceil(COUNT(*)::decimal / 16)
      FROM task_instance
      WHERE (state = 'running' OR state = 'queued')
        AND queue = 'default'

Three details make or break this. First, scale-to-zero is minReplicaCount: 0 — when the bookshop queue has been empty past the cooldownPeriod, KEDA removes the last worker and you pay for nothing overnight. The cooldownPeriod exists precisely so a brief lull between tasks doesn’t thrash the fleet down and back up; make it comfortably longer than your typical inter-task gap. Second, scoping the query by queue lets you run separate KEDA scalers per queue — a gpu worker Deployment scaling on the gpu queue’s depth, a default Deployment on the default queue’s — so expensive hardware only wakes when there’s expensive work. Third, scale-to-zero reintroduces a cold start on the first task after idle: the worker Deployment has to schedule a pod before anything runs. That’s fine for a nightly batch and wrong for a queue that must answer within seconds; keep minReplicaCount: 1 on the latter.

The honest way to think about KEDA-on-Celery versus KubernetesExecutor: both give you elastic, scale-to-zero cost, but they amortize startup differently. KEDA cold-starts a worker that then serves many tasks warm; KubernetesExecutor cold-starts a pod per task, forever. For a flood of small tasks, one warm worker spun up by KEDA beats a thousand pod creations. For a few big isolated tasks, per-task pods win. Which is, once again, the whole workload-shape argument — now with a dial on both sides.

The triggerer: deferrable tasks, independent of all of this

One component sits entirely outside the executor question and you must run it regardless of which executor you chose: the triggerer. When a task defers — a deferrable sensor waiting on a bookshop S3 file, a TimeDeltaTrigger, anything that calls self.defer(...) — it gives up its execution slot completely. It is not holding a Celery worker slot or a Kubernetes pod while it waits. Its trigger moves to the triggerer, a separate asyncio process that watches thousands of pending conditions on a single event loop, and only when the condition fires does the scheduler dispatch the task back to its executor to run the post-defer code.

That’s why deferrable beats classic sensors so decisively: a classic poke-mode sensor pins a whole worker (or, under KubernetesExecutor, a whole pod) doing nothing but sleeping and re-checking, while a deferred task costs a slice of one shared async loop. Under Kubernetes especially, the savings are stark — you are not paying pod-per-idle-wait.

For production, two triggerer facts matter. It’s horizontally scalable and highly available: run more than one triggerer and they partition the outstanding triggers between themselves, heartbeating so that if one dies its triggers are picked up by the survivors — no deferred task is stranded. And each triggerer has a capacity ([triggerer] default_capacity, default 1000 triggers) that caps how many conditions one instance watches at once; blow past it across your fleet and triggers queue waiting for a free slot. So sizing the triggerer is a real capacity-planning input, not an afterthought — if your bookshop graph fans out into thousands of concurrent deferrable sensors, you size and possibly replicate the triggerer for it. And the classic footgun stands: no triggerer running means deferred tasks defer and simply never wake up. The Astro CLI starts one for you locally; a hand-rolled deployment needs airflow triggerer as its own long-lived process.

KubernetesPodOperator is a different thing — and that’s the point

Here’s the distinction the chapter opened on, now that you have the machinery to see it. KubernetesExecutor is how the task process runs. KubernetesPodOperator is what a task does. They live at different layers, and confusing them is the single most common Kubernetes-and-Airflow mistake.

KubernetesExecutor runs every task as a pod, and that pod runs your Airflow task code — the @task function, the operator’s execute(). KubernetesPodOperator (KPO) is an operator: it’s a task whose entire job is to launch some other container — a Docker image that isn’t Airflow at all, maybe a Go binary, a dbt image, a model-training container — wait for it to finish, and collect its result. And crucially, KPO works under any executor. You can run KPO on LocalExecutor, on Celery, on Kubernetes. It doesn’t need KubernetesExecutor; it only needs credentials to a Kubernetes cluster to launch its container into.

from airflow.sdk import dag
from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator
from kubernetes.client import models as k8s


@dag(schedule="@daily", catchup=False)
def bookshop_ml_train():
    train = KubernetesPodOperator(
        task_id="train_recommender",
        name="recommender-trainer",
        namespace="ml-jobs",
        image="my-registry/recommender-trainer:2.4.0",  # NOT an Airflow image
        cmds=["python", "train.py"],
        arguments=["--epochs", "20", "--catalog", "s3://bookshop/catalog/"],
        container_resources=k8s.V1ResourceRequirements(
            requests={"cpu": "4", "memory": "16Gi", "nvidia.com/gpu": "1"},
            limits={"nvidia.com/gpu": "1"},
        ),
        node_selector={"accelerator": "nvidia-a100"},
        get_logs=True,
        on_finish_action="delete_pod",
    )
    train


bookshop_ml_train()

This DAG could be running on a plain Celery deployment with no KubernetesExecutor anywhere. The train_recommender task itself runs on a Celery worker; all it does is ask Kubernetes to launch the training container, stream its logs, and delete the pod when done. The training image doesn’t contain Airflow, doesn’t import your DAGs, and can be owned and versioned by a completely different team. That’s the real reason to reach for KPO: you want to run a container that isn’t your Airflow codebase, on Kubernetes, from any executor.

When to reach for which:

You want to…UseNotes
Run your normal Airflow task code, one isolated pod each, elastic costKubernetesExecutorExecution substrate; shape pods with pod_override
Run a foreign container (non-Airflow image, other team’s job) as a taskKubernetesPodOperatorWorks under any executor; needs cluster creds, not KubernetesExecutor
Run many small, similar, latency-sensitive tasks warmCeleryExecutorAmortizes startup; add KEDA for elastic cost
Fit everything on one host, keep ops minimalLocalExecutorSingle point of failure; fine for small/rerunnable
Cheap steady tasks and heavy isolated ones in one DAGMultiple executors + executor=Supersedes CeleryKubernetesExecutor; you operate both substrates

The two Kubernetes rows are the ones people conflate, so hold them apart: you can run KubernetesPodOperator tasks on a Celery fleet (no KubernetesExecutor in sight), and you can run KubernetesExecutor without ever writing a KubernetesPodOperator (your ordinary @task functions each get a pod). They compose freely because they answer different questions — how does the task run versus what does the task launch.

Examples in this chapter are docs-checked against the series’ stated versions (Apache Airflow 3.3.0), but they were not executed in this repository — treat the YAML and pod_override specs as unverified against a live cluster.

Final thoughts

Kubernetes is where Airflow stops being a scheduler with some workers and becomes a control plane that provisions compute on demand — and everything in this chapter is really about controlling that provisioning precisely. The pod lifecycle explains your latency and your debugging story. The base template plus pod_override explains how one task gets 16Gi and a tainted node while the task beside it stays small, with the container named base doing the load-bearing work of the merge. Multiple executors explain how the warm-worker jobs and the isolated-pod jobs finally live in one DAG without a hybrid hack, at the cost of operating both substrates. KEDA and the triggerer explain how you stop paying for idle — one scaling the workers to zero, the other holding thousands of deferred waits on a single loop that no executor ever notices.

The thread running through every chapter so far is the one this chapter ends on: Airflow gives you a clean separation between what your pipeline does and how and where it runs, and mastery is knowing which layer a given problem lives on — so you change the DAG when the logic is wrong, and change the deployment when the physics are. Pick the substrate that matches your workload’s shape, keep the DAG code oblivious to it, and you’ve bought yourself the freedom to change your mind about the hardest, most expensive decision in the stack without rewriting a single task.

Which leaves exactly one question unanswered, and it’s the one every chapter has quietly assumed away: who is allowed to do any of this? Every knob in this book — clearing a task, editing a connection, triggering a DAG through the API, writing a DAG file at all — is a permission someone either has or doesn’t. That’s the last chapter.

Next: Who Gets to Press the Buttons

Comments