Containerized Tasks: Run Anything, Blame Nothing

Your worker's Python is not your task's Python. DockerOperator and KubernetesPodOperator let a task bring its own universe — and hand you a debugging story that now spans two systems.

Sooner or later a task shows up that doesn’t want to live in your Airflow worker. The bookshop’s recommender needs torch, and torch wants a CUDA build. The finance team’s reconciliation script is written in Go. The image-resizing step depends on a fourteen-year-old C library whose maintainer has moved on to woodworking. Each is a perfectly reasonable piece of software and every one is a terrible houseguest: to run it inside your worker, you have to install it inside your worker — and now your orchestrator’s dependency tree is the union of every dependency tree in the company.

That union is where Airflow deployments go to die. Someone pins pandas<2 for the legacy loader, the new mart job needs pandas>=2.2, and two DAGs that have nothing to do with each other are suddenly in a fight because they share an interpreter. The fix isn’t a better resolver. The fix is to stop sharing the interpreter.

Containerize the task. Give it its own image, its own Python (or its own Go, or no Python at all), its own libraries, its own filesystem — and let Airflow do the thing it’s actually good at: start it, watch it, read the exit code, retry it. The container is the interface. Airflow never imports your code; it launches your image. This chapter is the two operators that do that, in full: DockerOperator when you have a daemon, KubernetesPodOperator when you have a cluster.

Why containerize a task at all

Dependency isolation is what gets you here. The worker’s Python is not your task’s Python, and pretending otherwise is a tax you pay forever. A containerized task’s dependencies are frozen into its image at build time — nothing the platform team does to the Airflow image can break it, and nothing you do can break the platform. That decoupling matters on any team where the people who own the orchestrator and the people who own the job are different people, which is most teams past about eight engineers.

Reproducibility is what saves you at 2am. registry.internal/bookshop/recommender:2.4 is a thing you can pull onto your laptop and run with the same command and the same environment — no “well, the worker had scikit-learn 1.4 back then.” A failed containerized task reproduces with a docker run, which beats reconstructing a worker’s site-packages from six weeks ago.

Language agnosticism unlocks work you’d otherwise refuse. Airflow is a Python orchestrator; nothing says a task has to be Python. A container is a process contract — here’s an image, here’s a command, here’s an exit code — so the Go reconciler and the crusty C pipeline are exactly as schedulable, retryable, and backfillable as a @task function.

The honest cost

Now the bill, because nobody puts this in the tutorial.

You need an image build: a Dockerfile, a CI job, a registry, a tagging convention, and a human answer to “how does a code change get from a pull request into the image the DAG names.” A task that used to be a function you edited is now an artifact you ship. On a small team that can genuinely be worse than a fat worker image.

You need registry auth on every machine that might pull, managed so it doesn’t rot. This is the Sunday failure: someone rotates the registry token, no DAG changes, and every containerized task starts failing at pull time with an error that looks nothing like a data bug.

You pay a cold start — pod scheduling plus a layer cache that isn’t warm, seconds to tens of seconds. Invisible on a 20-minute transform, ruinous on a thousand two-second tasks. Containers are a bad answer to “my tasks are tiny and there are a lot of them.”

And your debugging story now spans two systems. The task failed — but did the task fail, or did the launch? Bad exit code, image-pull error, OOMKill, evicted pod, a mount that didn’t exist on the host? Airflow only reports what it can see, and much of this chapter is about making sure it can see enough. “The container failed” is not a diagnosis, and the last section is entirely about turning it into one.

None of this should scare you off; it should tell you when. Containerize the task with awkward dependencies, the task another team owns, the task that isn’t Python, the task that needs hard resource limits. Don’t containerize the task that reads a config file and posts to Slack.

DockerOperator, knob by knob

DockerOperator runs a container on a Docker daemon and blocks until it exits. That’s the whole model, and it’s a good one when you have a Docker host — a VM running the Airflow worker with the daemon alongside it, a build box, a single-machine deployment.

from airflow.sdk import dag
from airflow.providers.docker.operators.docker import DockerOperator
from docker.types import Mount


@dag(schedule="@daily", catchup=False)
def bookshop_recommendations():
    score = DockerOperator(
        task_id="score_recommendations",
        image="registry.internal/bookshop/recommender:2.4",
        command="python score.py --date {{ ds }} --out /scratch/scores.parquet",
        docker_conn_id="internal_registry",
        environment={"MODEL_BUCKET": "s3://bookshop-models", "LOG_LEVEL": "info"},
        private_environment={"HF_TOKEN": "{{ conn.huggingface.password }}"},
        mounts=[Mount(source="/data/scratch", target="/scratch", type="bind")],
        mount_tmp_dir=False,
        network_mode="bookshop_net",
        docker_url="unix:///var/run/docker.sock",
        force_pull=True,
        auto_remove="success",
        cpus=2.0,
        mem_limit="4g",
        xcom_all=False,
    )
    score


bookshop_recommendations()

Every one of those arguments is doing work. Take them in order.

The command, and the entrypoint you didn’t override

command is templated, which is the whole reason {{ ds }} works there. A string is parsed shell-style; a list (["python", "score.py", "--date", "{{ ds }}"]) is passed as an argv vector with no shell involved, which is what you want the moment an argument could contain a space.

The subtlety people trip on: command overrides the image’s CMD, not its ENTRYPOINT. If the Dockerfile declares ENTRYPOINT ["python", "score.py"], then command="--date 2026-05-20" appends to it, and command="python score.py --date …" gets you python score.py python score.py --date …. There’s an entrypoint parameter to override that too. Nine times out of ten the confusion is an ENTRYPOINT you forgot about, and the symptom is a usage error from a binary you didn’t think you were invoking.

Getting config in

environment is a templated dict of env vars, and it’s the sanctioned channel for configuration. It’s also visible — those values render into the task’s Rendered Template view. Great for LOG_LEVEL, catastrophic for a token.

That’s what private_environment is for: same mechanism, but the values aren’t surfaced in the task detail or the logs. Anything from a connection or a Variable that would embarrass you in a screenshot belongs there. Above, {{ conn.huggingface.password }} pulls a secret out of Airflow’s connection store at render time and hands it to the container without ever writing it into the DAG file.

env_file points at a .env file whose pairs are loaded into the container; anything you also set in environment wins. Handy when a container has twenty settings — but note the file must exist where the operator runs (the worker), not on the Docker host, if those differ.

Getting files in: mounts, and the volumes you’ll see in old code

mounts takes docker.types.Mount objects — the modern, explicit form:

mounts=[
    Mount(source="/data/scratch", target="/scratch", type="bind"),
    Mount(source="bookshop_models", target="/models", type="volume", read_only=True),
]

type="bind" mounts a host path, type="volume" a named Docker volume, and read_only=True is a keyword instead of a :ro suffix you can typo.

What you’ll find in every pre-2022 StackOverflow answer is volumes=["/data/scratch:/scratch:rw"] — a list of colon-delimited strings. That parameter is deprecated, and in recent majors of the Docker provider it’s gone outright. If you’re porting a 2.x DAG forward and it explodes on an unexpected keyword argument, that’s your culprit.

mount_tmp_dir earns its own paragraph, because it’s the most common DockerOperator failure on a remote daemon. By default (True), the operator creates a temp directory on the machine running the operator and bind-mounts it into the container at /tmp/airflow, advertising it through an AIRFLOW_TMP_DIR env var. Beautiful when the worker and the daemon are the same host. When they’re not — a remote docker_url, Docker-in-Docker, a daemon on another box — the daemon is asked to bind-mount a path that exists only on the worker, and it fails with a mount error that explains none of this. Remote daemon? Set mount_tmp_dir=False.

Networking, and which daemon

network_mode puts the container on a network: "bridge" (the default), "host" (share the host’s stack — fast, and a security decision), "none", or a user-defined Docker network by name. That last form is what you want when the container must reach a sibling service: put the bookshop’s Postgres and the scoring container on bookshop_net and it can dial postgres:5432 like a normal citizen.

docker_url says which daemon. The default is the local socket, unix:///var/run/docker.sock, which the worker must have mounted and be permitted to use. Point it at a tcp:// URL and you’re driving a remote daemon — at which point tls_ca_cert / tls_client_cert / tls_client_key stop being decoration, because an unauthenticated Docker daemon on a TCP port is root on that host, offered to the internet.

Pull policy, cleanup, limits

force_pull=True re-pulls before every run. Without it, Docker uses whatever it cached for that tag — so re-pushing :2.4 with a fix and re-running the DAG cheerfully executes the old image while you spend an hour convinced the fix didn’t work. If your team mutates tags (most do, at least :latest and :dev), force_pull=True isn’t optional; with immutable git-SHA tags you can drop it and save the pull.

auto_remove is a string, not a boolean (the boolean form is deprecated): "never" (the default), "success", or "force". The production-sane setting is "success" — clean up the boring runs, keep the failures so you can docker inspect the corpse. Just remember “keep the failures” also means “accumulate the failures,” and something has to sweep them before they eat the host’s disk.

cpus and mem_limit cap the container, and they matter more than they look: without mem_limit, a runaway container can OOM the whole worker host and take every other running task with it. And skip_on_exit_code — the one most people miss — marks the task skipped rather than failed when the container exits with the code you name. That’s how you say “there was no new data today, and that’s fine” without inventing a sentinel file.

Registry auth, without a password in the DAG

DockerOperator takes no username and no password. It takes docker_conn_id, pointing at an Airflow connection of type Docker Registry — host is the registry URL, login and password are the credentials, and the operator logs in before it pulls. Set it once, keep the secret in Airflow’s encrypted connection store (or a secrets backend, per chapter 13), and every private pull just works. The credential never appears in the DAG file, never appears in the rendered template, and rotates in one place.

How a value escapes a container

This one is genuinely surprising, and people build systems on it without knowing what they built on.

When do_xcom_push is on (it is by default, inherited from BaseOperator), DockerOperator pushes an XCom. What does it push? The container’s stdout. The operator captures the log stream, and xcom_all decides how much of it becomes the value: False (the default) pushes only the last line; True pushes every line, as a list.

Read that again. The mechanism by which a containerized task returns a value to Airflow is: print it, last. No serialization contract, no return protocol, no typed boundary. The container’s final line of stdout becomes the string that lands in XCom and flows into the next task’s xcom_pull.

It works, it’s used everywhere, and it’s a trap laid for your future self. Add a logging.info("done!") at the end of score.py and you have silently changed your DAG’s data contract — the downstream task now gets "done!" instead of a row count. Turn on a library’s debug logging and a stray warning becomes your return value. The mechanism is invisible in the DAG, invisible in the image, and fails by producing plausible garbage rather than an error.

If you use it, use it deliberately: have the container print one machine-readable line — a JSON object — as the last thing it ever does, and treat that as the documented interface between image and DAG. Better still, use the other mechanism:

score = DockerOperator(
    task_id="score_recommendations",
    image="registry.internal/bookshop/recommender:2.4",
    command="python score.py --date {{ ds }}",
    retrieve_output=True,
    retrieve_output_path="/tmp/result.pickle",
)

With retrieve_output=True, the operator reads a pickled file out of the container’s filesystem at retrieve_output_path after it exits and pushes the unpickled object as the XCom. Now the return value is an explicit artifact your code chose to write, not an accident of whatever printed last. It costs you one pickle.dump in the image and buys you a return contract that a stray log line cannot corrupt.

The @task.docker shortcut

There’s a middle road worth knowing. The Docker provider ships a TaskFlow decorator that runs a plain Python function inside a container:

from airflow.sdk import dag, task


@dag(schedule="@daily", catchup=False)
def bookshop_scoring():
    @task.docker(image="registry.internal/bookshop/recommender:2.4", auto_remove="success")
    def score(ds=None):
        import torch  # not installed on the worker — installed in the image
        ...
        return {"scored": 1234}

    score()


bookshop_scoring()

Airflow serializes the function, ships it into the container, runs it there, and brings the return value back — container-level dependency isolation while you still write Python in the DAG file. The catch is that the function must be self-contained: its imports live in the image, not on the worker, and it’s easy to violate that boundary by accidentally closing over something from the DAG module. Lovely fit for “this one task needs a library the worker can’t have”; poor fit for “another team owns this image.”

KubernetesPodOperator, knob by knob

On a cluster, you don’t launch sibling containers — you launch pods. KubernetesPodOperator (everyone says KPO) creates a pod, waits for it, streams its logs, and cleans it up. It is the better-behaved cousin of DockerOperator in essentially every dimension, and it has more knobs because Kubernetes has more knobs.

from airflow.sdk import dag
from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator
from airflow.providers.cncf.kubernetes.secret import Secret
from kubernetes.client import models as k8s


@dag(schedule="@daily", catchup=False)
def bookshop_recommendations_k8s():
    score = KubernetesPodOperator(
        task_id="score_recommendations",
        name="bookshop-scorer",
        namespace="data-jobs",
        image="registry.internal/bookshop/recommender:2.4",
        cmds=["python", "score.py"],
        arguments=["--date", "{{ ds }}"],
        env_vars={"MODEL_BUCKET": "s3://bookshop-models"},
        secrets=[
            Secret("env", "HF_TOKEN", "recommender-secrets", "hf-token"),
            Secret("volume", "/etc/model-creds", "model-creds"),
        ],
        volumes=[
            k8s.V1Volume(
                name="scratch",
                empty_dir=k8s.V1EmptyDirVolumeSource(size_limit="20Gi"),
            )
        ],
        volume_mounts=[k8s.V1VolumeMount(name="scratch", mount_path="/scratch")],
        container_resources=k8s.V1ResourceRequirements(
            requests={"cpu": "500m", "memory": "2Gi"},
            limits={"cpu": "2", "memory": "8Gi"},
        ),
        image_pull_policy="Always",
        image_pull_secrets=[k8s.V1LocalObjectReference(name="regcred")],
        node_selector={"workload": "batch"},
        tolerations=[
            k8s.V1Toleration(key="dedicated", operator="Equal",
                             value="batch", effect="NoSchedule")
        ],
        startup_timeout_seconds=300,
        get_logs=True,
        log_events_on_failure=True,
        on_finish_action="delete_succeeded_pod",
        in_cluster=True,
        do_xcom_push=True,
    )
    score


bookshop_recommendations_k8s()

cmds and arguments, not command

Kubernetes splits the command in two, and KPO mirrors it faithfully. cmds maps to the container’s command in the pod spec, which overrides the image’s ENTRYPOINT. arguments maps to args, which overrides the image’s CMD. Both are lists, both are templated.

Which means the mental model is inverted from Docker’s, and mixing them up is the classic first-KPO bug. If the image has ENTRYPOINT ["python", "score.py"] and you want to pass a date, you set only arguments=["--date", "{{ ds }}"] and leave cmds alone. Set cmds=["python", "score.py"] as well and you’ve replaced the entrypoint with the same thing, which happens to work — until the image’s entrypoint changes and your DAG doesn’t notice.

Where the pod lives, and how Airflow reaches the cluster

namespace is where the pod is created. Give the bookshop’s jobs their own namespace with a ResourceQuota and a LimitRange, and a bad DAG can’t starve the cluster.

in_cluster=True says “I am running inside Kubernetes; use my pod’s own ServiceAccount token to talk to the API server.” That’s the setting when Airflow itself runs on the cluster it launches into — no kubeconfig, no stored credential, just RBAC on a service account.

kubernetes_conn_id (default kubernetes_default) is the other path: Airflow runs somewhere else — a VM, a Celery fleet on EC2, your laptop — and reaches the cluster through a connection carrying a kubeconfig. That pairing is the thing that makes KPO-vs-KubernetesExecutor click, and we’ll get there shortly. config_file and cluster_context point at a kubeconfig on disk and pick a context in it; that’s the local-dev shape.

service_account_name sets the ServiceAccount the launched pod runs as — distinct from the identity Airflow used to create it. That’s how the scoring pod gets an IRSA / Workload-Identity binding to read the model bucket without any cloud keys existing anywhere.

Config and secrets

env_vars is a templated dict (or a list of k8s.V1EnvVar if you need valueFrom), and it’s visible.

secrets is the good part. A Secret binds a Kubernetes Secret into the pod; its positional arguments are deploy_type, deploy_target, secret, key:

  • Secret("env", "HF_TOKEN", "recommender-secrets", "hf-token") reads the hf-token key out of the recommender-secrets Secret and exposes it as the HF_TOKEN environment variable.
  • Secret("volume", "/etc/model-creds", "model-creds") — no key — mounts every key of that Secret as files under /etc/model-creds. That’s how a job gets a credentials.json without anyone base64-ing it into a DAG.

The value never passes through Airflow. Airflow references the Secret by name; the kubelet resolves it. Your DAG file, your rendered templates, and your task logs contain the name of a secret, not a secret. That’s a materially better posture than environment={"TOKEN": "…"}, and it’s one of the strongest arguments for KPO over DockerOperator on a cluster. env_from (a list of k8s.V1EnvFromSource) is the bulk version — pull a whole ConfigMap or Secret in as env vars in one line.

Volumes and resources

volumes and volume_mounts are the raw Kubernetes objects in two lists, joined by name: a V1Volume declares what (an emptyDir, a persistentVolumeClaim, a configMap), a V1VolumeMount declares where in the container. The emptyDir above is the standard “give this pod scratch space that dies with it” move, with a size_limit so a runaway job fills its own disk instead of the node’s.

container_resources takes a k8s.V1ResourceRequirements. Requests are what the kube-scheduler uses to place the pod — promise too much and it sits Pending forever with FailedScheduling; promise too little and you land on a node that’s already full. Limits are what the kubelet enforces — exceed the memory limit and the kernel OOMKills your container, which is the failure mode we dissect at the end of this chapter.

The gotcha is the parameter name: it is container_resources, not resources. resources is a BaseOperator argument about Airflow’s own pool accounting, and passing a V1ResourceRequirements there is a hard error rather than a silent one — TypeError: Resources() argument after ** must be a mapping, not V1ResourceRequirements. That’s the good outcome: you find out at parse time. The bad outcome is the opposite mistake — omitting container_resources entirely, which is silent, and leaves your pod inheriting the namespace’s default limits. If your pod keeps inheriting the namespace’s default limits no matter what you set, check the keyword.

Pulling the image

image_pull_policy is the Kubernetes cousin of force_pull: "Always" re-pulls for every pod, "IfNotPresent" uses the node’s cache, "Never" demands it already be there. Same reasoning — mutable tags need "Always", immutable SHAs don’t.

image_pull_secrets takes k8s.V1LocalObjectReference(name="regcred") objects naming a dockerconfigjson Secret in the pod’s namespace. Note where the credential lives: in the cluster, not in Airflow. Airflow just names it. (Most platform teams skip the parameter entirely and attach the pull secret to the namespace’s ServiceAccount — then every pod can pull and no DAG has to know.)

startup_timeout_seconds (default 120) is how long the operator waits for the pod to leave Pending before giving up. On a cold node pulling a 6GB CUDA image, 120 seconds is not enough, and the resulting Pod took too long to start reads like a bug in your job when it’s really a bug in your patience. Raise it for fat images.

Cleanup, and one dead knob that looks alive

on_finish_action decides the pod’s fate:

  • "delete_pod" — the default. Always delete, success or failure.
  • "keep_pod" — never delete. You now own a growing pile of Completed pods.
  • "delete_succeeded_pod" — delete on success, keep on failure. This is the setting you want in production: successes vanish, failures stick around for kubectl describe.

And now the sharp edge. The old parameter was is_delete_operator_pod, a boolean. It is still accepted by the operator’s constructor — passing it raises no error, no TypeError, nothing. It is also no longer read: the operator derives its cleanup behaviour entirely from on_finish_action, whose default is delete_pod. So a DAG that says

    is_delete_operator_pod=False,   # dead config — looks load-bearing, isn't

reads, to every human who opens it, like a task that keeps its pod for debugging. It does not. It deletes the pod, every time, because on_finish_action defaulted to delete_pod and nobody overrode it. This is the worst possible category of bug: config that is syntactically fine, semantically inert, and misleading to the reader. If you’re porting a 2.x DAG, grep for is_delete_operator_pod, delete every occurrence, and set on_finish_action to what you actually meant.

Placement: node selectors, tolerations, affinity

node_selector is the blunt instrument — a dict of labels the node must have. {"workload": "batch"} keeps the scoring pod off the nodes running your API.

tolerations let a pod land on a tainted node, which is the half of the dedicated-node-pool pattern people remember. Here’s the half they forget: a toleration only grants permission to land on the taint, it doesn’t attract the pod there. You almost always need a node_selector (or node affinity) alongside it — and forgetting that is why your expensive GPU pool sits empty while the GPU job runs happily on a general node with no GPU in it.

affinity is the expressive form: V1Affinity with node/pod/anti-pod rules, preferred (soft) as well as required (hard). It’s the tool for “spread these across zones” or “never co-schedule two of these.” It’s also verbose enough that it belongs in a module-level constant, not inline in the DAG.

do_xcom_push, and the sidecar you didn’t ask for

KPO’s XCom mechanism is different from Docker’s, and — unlike Docker’s — it does not read stdout. When you set do_xcom_push=True (the default is False; the pod pushes nothing unless you ask):

  1. The operator adds an emptyDir volume to the pod, mounted at /airflow/xcom in your container.
  2. It injects a sidecar container — a tiny alpine image — sharing that same volume, whose only job is to sit there doing nothing.
  3. Your container runs, and before it exits it writes a JSON file to /airflow/xcom/return.json.
  4. Your container exits. The pod does not die, because the sidecar is still running.
  5. Airflow execs into the sidecar, reads /airflow/xcom/return.json, pushes it as the XCom, and tells the sidecar to exit. The pod finishes.

A shared emptyDir, a file, and a babysitter container that exists purely to keep the pod alive long enough for Airflow to read the file — because once every container in a pod has exited, the pod’s filesystem is gone. The consequences fall right out of it:

  • The file must be valid JSON. print() does nothing; your container has to actually write it.
  • Write it unconditionally. A pod that succeeds but forgets the file fails at the very last step of an otherwise-green task. If you have nothing to return, write {}.
  • The directory must be writable by whatever user the image runs as. A hardened image with a read-only root filesystem fails here for reasons that have nothing to do with your code.
  • The sidecar is really in the pod. It shows up in kubectl get pod, it nibbles at the namespace quota, and a policy engine that rejects pods with unexpected containers will reject yours.

The distinction that trips everyone: operator vs. executor

Two things here are named almost identically and are not the same thing, and it is the single most common Kubernetes-and-Airflow confusion.

KubernetesPodOperator is an operator — something you put in a DAG, saying “this task’s job is to launch a container.” It works under any executor: Local, Celery, Kubernetes, Edge. It only needs credentials to a cluster.

KubernetesExecutor is an executor — something your platform team sets in config, saying “every task in this deployment runs in its own pod.” It has nothing to do with what your tasks do.

KubernetesPodOperatorKubernetesExecutor
What it isAn operator you put in a DAGAn executor set in deployment config
Who chooses itThe DAG author, per taskThe platform team, per deployment
What runs in a podThe container this one task launchesEvery task, whatever operator it uses
What’s in the imageYour job — Go, R, CUDA, anythingAirflow — it runs airflow tasks run
Works under any executor?Yes — Local, Celery, KubernetesIt is the executor
Per-task shapingOperator kwargs (container_resources, …)executor_config={"pod_override": …}
Needs a cluster?Yes, to launch intoYes, to run on

The line that clarifies it: KubernetesExecutor decides how your task process runs. KubernetesPodOperator decides what your task launches. They live at different layers, and they compose freely — you can run KPO tasks on a plain Celery deployment with no KubernetesExecutor anywhere in sight (the Celery worker runs the operator, which asks the cluster for a pod), and you can run KubernetesExecutor and never write a single KPO (your ordinary @task functions each get a pod).

You can also do both at once, which is where the mental model earns its keep: under KubernetesExecutor, a KPO task means Airflow creates a pod (to run the task process) which then creates another pod (the job). That’s occasionally exactly right — you want the job’s image to be independent and the task process isolated — and it’s often a sign you could have picked one. Chapter 05 covers the executor choice; chapter 17 goes all the way down into pod templates, pod_override merge semantics, and multi-executor deployments. This chapter is only about the operator.

And which container operator should you use?

Mostly, whichever your compute already is. If you have a Docker daemon and no cluster, DockerOperator is the entire answer and it’s a good one. If you’re on Kubernetes, KPO is strictly the better citizen: real requests and limits, native Secret mounting, RBAC per namespace, node placement, and cleanup the cluster handles.

The anti-pattern to name explicitly: running DockerOperator on a Kubernetes deployment by bind-mounting the node’s Docker socket into an Airflow worker pod. It works in a demo. In production it means any DAG author can start a privileged container on a node, bypassing every Kubernetes control you have — resource quotas, network policy, pod security, the scheduler itself. It’s a container escape with extra steps. On a cluster, launch pods.

Deferrable pods: the six-hour job that holds nothing

By default, a KPO task occupies a worker slot for the entire life of its pod. The operator sits in a loop watching pod status. If the pod runs for six hours, an Airflow worker slot is pinned for six hours doing nothing but asking Kubernetes “is it done yet?”

deferrable=True fixes that:

    score = KubernetesPodOperator(
        task_id="score_recommendations",
        image="registry.internal/bookshop/recommender:2.4",
        ...
        deferrable=True,
        poll_interval=10,
    )

The operator creates the pod, calls self.defer(...), and releases its execution slot entirely. The watching moves to the triggerer — the async process from chapter 02 — which babysits thousands of pending pods on one event loop, because “poll the Kubernetes API every ten seconds” is exactly the cheap await that asyncio was built for. When the pod reaches a terminal phase, the trigger fires an event and the scheduler puts the operator’s completion method back on a worker to collect the result and clean up.

The economics are stark. Twenty long-running pods under a non-deferrable KPO pin twenty worker slots — and under KubernetesExecutor, twenty pods whose entire purpose is watching twenty other pods. Deferred, they pin zero. The costs: you must actually be running a triggerer (airflow triggerer; the Astro CLI starts one), and logs arrive differently — in deferred mode the trigger fetches container logs on an interval rather than streaming continuously, so the UI fills in chunks, with logging_interval controlling the cadence. That’s why some teams leave short pods non-deferrable and reserve deferrable=True for the long ones. The heuristic is right: defer when the wait is long or the fan-out is wide.

”How do I know it worked?”

The question the whole chapter has been building to, and it deserves a precise answer, because containerized tasks fail in ways ordinary tasks can’t.

The exit code is the contract

Both operators key success off the container’s exit status. Zero succeeds; non-zero fails the task and hands it to Airflow’s retry machinery. That is the cleanest signal in this entire book — the thing that did the work is the thing whose result Airflow reads. No polling, no external API, no eventual consistency.

Which means your image owes Airflow an honest exit code, and a shocking number of images don’t give one. A Python script that catches every exception and logs it exits zero. A shell pipeline (extract | transform) exits with the status of the last command, so a failing extract feeding a happy transform is a green task with no data. A set -e-less bash entrypoint marches cheerfully past its own failures. If you take one operational lesson from this chapter: audit your images’ exit codes. A containerized task that can’t fail is not a task, it’s a rumour.

Two codes to memorize, because you’ll read them in pod status: 137 is 128 + 9, SIGKILL — on Kubernetes that almost always means OOMKilled. 143 is 128 + 15, SIGTERM — something asked it to stop politely, usually an eviction or a kubectl delete.

Getting the logs back

get_logs=True (KPO’s default) streams the container’s stdout/stderr into the Airflow task log. That’s not a nicety. With on_finish_action="delete_pod", the pod is gone by the time you look and kubectl logs has nothing to show you — the only copy of the container’s output is the one Airflow pulled while it was running. Turn get_logs off and a failed pod becomes a red box with no explanation. DockerOperator streams logs the same way, which is also, awkwardly, why its stdout doubles as its XCom channel.

The failures that aren’t in the logs

And now the part that separates people who’ve run this in production from people who’ve read the docs.

OOMKilled. Your container exceeded its memory limit. The kernel killed it — instantly, SIGKILL, no traceback, no exception, no “MemoryError.” The task log just ends mid-sentence. Exit code 137. There is nothing in the logs explaining it, because the process was never given a chance to say anything. The evidence lives in the pod’s container status (reason: OOMKilled) and its events, not in your application’s output.

Evicted. The node ran out of memory or disk and the kubelet threw your pod overboard to save itself. Phase Failed, reason: Evicted — and here’s the cruel part: an evicted pod may have its container filesystem reclaimed, so even the logs can be gone. This is what happens when your memory request sat far below actual usage: the scheduler packed the node based on your optimism, the node filled, and the kubelet started evicting. The fix is not “retry.” The fix is an honest request.

FailedScheduling. The pod never started. It sat Pending until startup_timeout_seconds expired, because no node had room for its requests, or its node_selector matched nothing, or it tolerated a taint with nothing attracting it to the pool. Empty logs, because there was never a container.

ImagePullBackOff. The registry said no — wrong tag, expired pull secret, secret in the wrong namespace. Again: Pending pod, empty logs, timeout.

Every one of these presents identically in Airflow’s UI — a failed task with useless or absent logs — and every one has a completely different fix. The evidence is in Kubernetes events, and one parameter drags it into the place you’re already looking:

    log_events_on_failure=True,

Set it. When a pod fails, KPO dumps its Kubernetes events into the task log, and OOMKilled / Evicted / FailedScheduling / ImagePullBackOff show up in the Airflow UI instead of requiring cluster access and a race against the garbage collector. Pair it with on_finish_action="delete_succeeded_pod" so failed pods survive for a kubectl describe, and you’ve turned “the container failed, I guess” into an actual diagnosis. Its sibling log_pod_spec_on_failure dumps the resolved pod spec too — invaluable when the real bug is that your container_resources never applied the way you thought.

On the Docker side the equivalent discipline is auto_remove="success" plus the habit of docker inspect-ing a failed container’s State for OOMKilled: true. Docker will tell you — you just have to ask, and you have to have kept the corpse.

Retries, and the pod that’s still running

One last piece of machinery. reattach_on_restart (default True) handles the case where the Airflow side dies while the pod is fine: the worker is killed, the task is queued for retry, and the pod is still happily grinding away. On the retry, rather than launching a second pod to do the same work, KPO finds the pod carrying that exact task instance’s labels and reattaches — resuming the log stream and waiting for the original. It quietly saves you from double-running an expensive job every time a worker gets rescheduled, and it means your pods’ labels are load-bearing. Don’t fight the operator over them.

Final thoughts

Containerizing a task is a trade, and it’s worth being clear-eyed about both sides. You buy dependency isolation, reproducibility, and the freedom to orchestrate software that has never heard of Python — genuinely large wins, and the only real answer once more than one team’s code has to run on the same orchestrator. You pay with a build pipeline, a registry, a cold start, and a failure surface that now includes the container runtime, the kube-scheduler, the node, and the registry — none of which write anything useful into your application’s logs.

You make that trade pay by closing the visibility gap on purpose. Give your image an honest exit code, because that’s the only contract Airflow actually checks. Turn on get_logs so the container’s output survives the pod. Turn on log_events_on_failure so the cluster’s explanation lands in the same place. Delete successful pods, keep the failed ones. Set a memory request you believe rather than one you hope for. Use Secret objects instead of environment variables and let the kubelet handle the credential. Defer the long ones so a six-hour pod costs a coroutine instead of a worker. And write /airflow/xcom/return.json deliberately — or better, don’t return anything at all, because a task that writes its output to object storage and returns nothing has no XCom to corrupt.

Get that right and a containerized task is the most honest kind of task there is: a black box with a clean interface, an image you can run on your laptop, and an exit code that means what it says. Airflow starts it, watches it, and gets out of the way — which, as the next chapter argues at length, is the only thing Airflow should ever be doing.

Next: Delegating Compute: Airflow Is the Conductor, Not the Orchestra

Comments