Executors: Local, Celery, and Kubernetes
The scheduler decides what's ready to run; the executor decides how and where it actually runs. Pick the wrong one and you feel it.
The scheduler gets the glory, but it never runs a single line of your task code. Its whole job is bookkeeping: it looks at the DAGs, works out which task instances have their dependencies met, and marks them ready. Then it hands them off. The thing on the other side of that handoff — the component that actually finds a machine, spins up a process, and runs your Python — is the executor. Same DAG, same scheduler; swap the executor and you’ve changed how your entire pipeline scales, isolates, and costs.
You pick an executor in config, and — this is the headline change in 3.x — you no longer have to pick just one. But the DAG code still doesn’t know or care which executor it runs under. That separation is the whole point, and it’s what lets everything below be a deployment decision rather than a rewrite.
LocalExecutor
The simplest thing that parallelizes. LocalExecutor runs each task as a subprocess on the same host as the scheduler.
AIRFLOW__CORE__EXECUTOR=LocalExecutorThat’s the entire setup — no broker, no worker fleet, no extra moving parts. It’s what astro dev start gives you locally, and it’s a perfectly honest choice for small production too: if your workload fits on one reasonably-sized box, a single host running dozens of concurrent tasks is far less to operate than anything distributed.
The knob that matters is parallelism, the ceiling on how many task subprocesses run at once across the whole deployment:
AIRFLOW__CORE__PARALLELISM=32The default is 32. Every ready task spawns a subprocess, and when 32 are live the rest queue until one exits — so parallelism is really a promise about how much concurrent memory the host can survive. Set it by dividing available RAM by a task’s realistic peak, not by wishful thinking; a bookshop ingestion task that pulls a day of orders into memory needs very different headroom than a task that just triggers dbt and waits. There are two other ceilings stacked above it — max_active_tasks_per_dag (per DAG) and each pool’s slot count — and the effective limit is always the smallest of the three that applies.
The ceiling is exactly what it sounds like: one machine’s CPU and memory, and one machine’s uptime. LocalExecutor is a single point of failure — the scheduler host is the worker fleet, so if it reboots, every running task dies with it and there is nowhere else for the work to go. That’s fine for a nightly bookshop refresh you can rerun; it’s not fine for a deployment that has to keep making progress through a node failure. When tasks queue because the host is saturated, or when a single host’s downtime is unacceptable, you’ve outgrown it, and no config knob fixes either problem.
The SequentialExecutor is gone. If you’re arriving from a 2.x tutorial, you may remember
SequentialExecutoras the SQLite-backed default that ran one task at a time. It was removed in Airflow 3 — it existed only because SQLite couldn’t handle concurrent writes, and the modern local setup ships with Postgres instead. LocalExecutor with a real metadata database is now the floor. Don’t go looking for the sequential one; there’s nothing to configure.
CeleryExecutor
The classic way past a single host. CeleryExecutor puts a broker — Redis or RabbitMQ — between the scheduler and a pool of long-running worker processes. The scheduler drops ready tasks onto the queue; workers pull them off and run them. Add capacity by adding workers.
AIRFLOW__CORE__EXECUTOR=CeleryExecutor
AIRFLOW__CELERY__BROKER_URL=redis://redis:6379/0
AIRFLOW__CELERY__RESULT_BACKEND=db+postgresql://airflow:airflow@postgres/airflowTwo pieces of infrastructure, not one. The broker carries the messages — “here is a task to run” — from scheduler to workers. The result backend carries the outcomes back; Celery workers report task state through it, and the standard, boring choice is to point it straight at your existing Airflow metadata database with the db+ prefix as shown above. People sometimes reuse Redis as both broker and result backend to save a component; resist it for anything real, because a Redis eviction or restart then loses task results as well as queued messages.
The workers are always on, so there’s no per-task startup cost — throughput is excellent and latency is low. The trade you’re making is warmth for elasticity: capacity is whatever you’ve provisioned, so you pay for idle workers in the troughs and you can starve at the peaks unless you autoscale.
Worker concurrency
Each worker runs several tasks at once. How many is worker_concurrency:
AIRFLOW__CELERY__WORKER_CONCURRENCY=16That’s 16 task processes per worker. Total cluster capacity is worker_concurrency × number of workers, capped by the same parallelism ceiling from before — set parallelism too low and your beautiful worker fleet sits half-idle because the scheduler won’t dispatch past the global limit. Because Airflow tasks are usually I/O-bound — they trigger a dbt run, submit a Spark job, poll an API, and mostly wait — you can safely run concurrency well above the worker’s CPU count. A worker with 4 vCPUs happily handles 16 tasks that spend their lives waiting on other systems.
For workloads that swing between quiet and busy, worker_autoscale lets a single worker grow and shrink its own process count between a floor and a ceiling:
AIRFLOW__CELERY__WORKER_AUTOSCALE=16,4Up to 16 when there’s work, down to 4 when there isn’t — the max comes first. This is process-level autoscaling within a worker; it doesn’t add or remove worker machines. For that you want KEDA, below.
Queues: routing work to the right workers
By default every worker pulls from one shared queue. That’s wasteful the moment your tasks aren’t uniform — you don’t want a heavyweight GPU job and a two-second API poll competing for the same processes. Queues fix this. Tag a task with the queue it belongs on:
from airflow.sdk import dag, task
@dag(schedule="@daily", catchup=False)
def bookshop_ml():
@task(queue="gpu")
def train_recommender():
# runs only on workers listening to the "gpu" queue
...
@task # default queue
def refresh_sales_mart():
...
refresh_sales_mart() >> train_recommender()
bookshop_ml()Then start workers that listen only to the queues they’re built for. Your GPU boxes subscribe to gpu; everything else stays on the default:
# on the GPU node
airflow celery worker -q gpu
# on a general-purpose node
airflow celery worker -q defaultNow train_recommender can only ever land on a GPU worker, and refresh_sales_mart never wastes one. Queues are how you keep a single Celery fleet heterogeneous without letting the wrong task type saturate the wrong hardware. The default queue name comes from AIRFLOW__OPERATORS__DEFAULT_QUEUE (it’s default), and a task’s queue has to match a queue some worker is actually listening on — tag a task gpu with no gpu worker running and it queues forever, silently.
KEDA: autoscaling the worker fleet
worker_autoscale grows processes inside a fixed set of machines; KEDA grows the machines. Running Celery on Kubernetes (the Airflow Helm chart supports this directly), KEDA watches a metric and scales the worker Deployment against it. The right metric is the depth of the queue — specifically a SQL query against the metadata DB counting queued and running task instances:
workers:
keda:
enabled: true
minReplicaCount: 1
maxReplicaCount: 10
# scale one worker per this many queued/running tasks
query: >-
SELECT ceil(COUNT(*)::decimal / 16)
FROM task_instance
WHERE (state='running' OR state='queued')The 16 is your worker_concurrency: one worker per sixteen pending tasks. When the bookshop DAG fires at midnight and floods the queue, KEDA spins workers up toward maxReplicaCount; when the backlog drains, it scales back to minReplicaCount — often 1, or even 0 for a fleet that can tolerate cold starts. This is how Celery reclaims the elasticity that Kubernetes gets for free: you keep warm-worker latency during the day and stop paying for a full fleet overnight.
Flower, and the Redis visibility-timeout trap
Celery ships a monitoring UI called Flower — a live view of workers, queues, and task throughput — which you can run alongside the deployment (airflow celery flower, or the Helm chart’s flower.enabled). It’s handy for watching a fleet, though most day-to-day debugging still happens in Airflow’s own UI.
There’s a robustness detail here that’s easy to miss: Celery decouples worker lifetime from scheduler lifetime. Because tasks live on the broker and run in independent worker processes, a scheduler restart doesn’t kill in-flight work — the running tasks keep going, and when the scheduler comes back it adopts them, reconciling their state from the workers rather than starting over. That’s the opposite of LocalExecutor, where the scheduler host and the worker are the same process and a restart takes everything down with it. It’s a big part of why Celery is the workhorse for deployments that have to survive a rolling restart mid-run.
The one Celery gotcha worth burning into memory involves Redis as a broker. Redis has no native concept of “this message is being worked on,” so Celery fakes it with a visibility timeout: when a worker picks up a task, the message is hidden for a fixed window, and if it isn’t acknowledged as done within that window, Redis makes it visible again and another worker picks it up. The default window is one hour. So any Airflow task that runs longer than the visibility timeout can be dispatched a second time while the first copy is still running — a duplicate, concurrent execution of the same task instance. For the bookshop this is mostly a nuisance; for a task that writes to a warehouse or moves money it’s a correctness bug. The fix is to set the timeout comfortably above your longest task:
AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__VISIBILITY_TIMEOUT=21600Six hours here. RabbitMQ doesn’t have this problem — it acknowledges messages properly — which is a real reason to prefer it as a broker if your tasks are long-running.
KubernetesExecutor
Where Celery keeps warm workers waiting, KubernetesExecutor launches one pod per task and tears it down when the task finishes. Nothing idles.
AIRFLOW__CORE__EXECUTOR=KubernetesExecutorTwo things fall out of that model. First, isolation: every task gets its own pod, its own image, its own resource limits. A memory-hungry Spark submit and a tiny API poll no longer share a host or fight over a worker’s RAM, and one task’s crash can’t touch another’s. Second, elasticity: pods exist only while tasks run, so your cluster scales with actual demand and — paired with a cluster autoscaler — you’re not paying for a worker pool at 3am. The cost is latency: pod scheduling and image pull add seconds to every task, invisible on a 20-minute transform and painful on a thousand two-second tasks. This is the executor for spiky, heterogeneous, resource-hungry workloads that benefit from hard isolation.
The lever that makes per-task isolation real is executor_config, which lets a single task override the pod it runs in — a bigger memory limit, a different image, a GPU request, a node selector:
from kubernetes.client import models as k8s
@task(
executor_config={
"pod_override": k8s.V1Pod(
spec=k8s.V1PodSpec(
containers=[
k8s.V1Container(
name="base",
resources=k8s.V1ResourceRequirements(
requests={"memory": "4Gi"},
limits={"memory": "8Gi"},
),
)
]
)
)
}
)
def rebuild_search_index():
...pod_override is a full Kubernetes V1Pod that Airflow merges over the base worker template, so you can reshape one task’s runtime without touching any other. That’s the whole superpower of this executor, and it deserves more room than an overview gives it — the base pod template, the merge semantics, affinity and tolerations, secrets and volumes, and the sharp edges all live in Chapter 17, on Kubernetes and multi-executor deployments. Here, just know the seam exists: the executor gives every task a pod, and executor_config is how you make that pod fit the task.
Celery vs. Kubernetes: the actual trade-off
Both scale past a single host, so the choice between them is really a choice between warm capacity and per-task isolation. Laid side by side:
| Dimension | CeleryExecutor | KubernetesExecutor |
|---|---|---|
| Startup latency | ~none — workers are warm, tasks start instantly | seconds per task — pod schedule + image pull |
| Throughput | excellent for many short tasks | pod-creation rate becomes the bottleneck at high fan-out |
| Isolation | tasks share a worker’s OS, RAM, and Python env | one pod per task — separate image, limits, blast radius |
| Idle cost | pay for the worker fleet even when quiet (unless KEDA) | near-zero — pods exist only while tasks run |
| Per-task resources | coarse, via queues → specialized worker pools | fine, via executor_config/pod_override per task |
| Scaling model | add workers / KEDA on queue depth | cluster autoscaler adds nodes on pod demand |
| Ops surface | broker + result backend + worker fleet to keep healthy | a Kubernetes cluster to run |
| Classic failure mode | Redis visibility-timeout double-run; broker outage stalls dispatch | image-pull failures; pod scheduling latency under load |
The short version: lots of small, similar, latency-sensitive tasks favor Celery, because warm workers amortize the startup cost that Kubernetes pays on every single task. Heterogeneous, bursty, resource-hungry tasks that want hard isolation favor Kubernetes, because per-pod limits and elastic cost are exactly what a uniform worker fleet can’t give you. Most of the pain people attribute to “Airflow doesn’t scale” is really a workload sitting on the wrong side of this table.
You don’t have to choose just one
Historically that table forced an awkward compromise. Teams with both profiles — a flood of cheap sensors and a few monstrous Spark jobs — reached for the old CeleryKubernetesExecutor, a bolted-together hybrid that routed tasks to Celery or Kubernetes based on a magic queue name. It worked, but it was one more thing to understand and operate.
Airflow 3 replaces it with something cleaner: multiple executors, configured as a list, chosen per task. You declare every executor the deployment should support:
AIRFLOW__CORE__EXECUTOR=CeleryExecutor,KubernetesExecutorThe first entry is the default — any task that doesn’t say otherwise runs on Celery. Then any individual task opts into a different one with executor=:
from airflow.sdk import dag, task
@dag(schedule="@daily", catchup=False)
def bookshop_nightly():
@task # default → CeleryExecutor, warm and fast
def poll_orders_api():
...
@task(executor="KubernetesExecutor") # isolated, big, its own pod
def rebuild_search_index():
...
poll_orders_api() >> rebuild_search_index()
bookshop_nightly()Now the cheap poll rides a warm Celery worker with near-zero startup, and the memory-hungry index rebuild gets its own Kubernetes pod with its own limits — in the same DAG, from the same scheduler. You can even reference executors by an alias defined in config to keep the strings tidy. This supersedes CeleryKubernetesExecutor entirely; the hybrid is deprecated, and the right mental model now is “a deployment has a list of executors, and each task picks from it.” Pick the default for the shape most of your tasks have, and reach for executor= on the exceptions.
The triggerer runs deferrable tasks — no matter which executor
There’s a component here that sits outside the whole executor discussion, and it trips people up: the triggerer. When a task defers — a deferrable sensor waiting for a file, TimeDeltaTrigger waiting out a delay, any task that hits self.defer(...) — it does not hold a worker, a subprocess, or a pod while it waits. It releases its execution slot entirely, and its trigger is handed to the triggerer, a separate async (asyncio) process that can watch thousands of pending conditions at once on a single thread. When the condition fires, the triggerer wakes the task and the scheduler dispatches it back to whichever executor it belongs to, to run its post-defer code.
The reason this matters in an executors chapter: the triggerer’s role is identical regardless of executor. Local, Celery, Kubernetes, a mix — deferred waiting always happens on the triggerer, never on a worker or a pod. That’s precisely why a deferrable sensor is so much cheaper than a classic one: a classic sensor in poke mode occupies a full worker slot (or a whole pod, under Kubernetes) doing nothing but sleeping and re-checking, while a deferrable sensor occupies a shared async loop. This is the payoff of the deferrable-operators work from the sensors chapter — but it only exists if a triggerer is actually running. The Astro CLI starts one for you; a hand-rolled deployment needs airflow triggerer as its own process, and forgetting it means deferrable tasks defer and simply never come back.
The Edge executor
The executors above assume workers sit next to the scheduler on the same network. The Edge executor, newer in 3.x, drops that assumption: workers talk to the api-server over HTTP, so they can live anywhere you can reach a URL from. A worker in another datacenter, in a customer’s VPC, on-prem next to a database that can’t be exposed — it pulls work over the API and runs it where the data actually is.
That HTTP model changes the security story. There’s no broker to share, no direct database connection from the worker back to the core — the worker authenticates to the api-server with a shared secret (a JWT signed with a configured key), polls for work assigned to its queue, and reports results back over the same authenticated channel. Because the connection is worker-initiated and outbound, the remote site only needs to reach your control plane, not be reachable by it — no inbound firewall holes into the customer’s VPC. That’s a different problem from raw scale: Edge is about reach — running tasks in places the scheduler can’t see directly — and it’s how Airflow starts to look like a distributed control plane rather than a single cluster. If your compute has to run somewhere your control plane doesn’t, this is the one to read up on.
Choosing, and not having to
The honest decision tree is short. One host is enough → LocalExecutor. Need horizontal scale, lots of similar warm-friendly tasks, and you’re comfortable operating a broker → CeleryExecutor. Want per-task isolation and elastic cost on a cluster you already run → KubernetesExecutor. Both profiles at once → list both and route with executor=. Workers have to run somewhere remote → Edge. And whichever you pick, a triggerer runs alongside it so deferred tasks wait cheaply.
And if operating any of this sounds like a second job, that’s because it is — which is exactly what the managed platforms sell. Astro, MWAA, and Composer pick an executor, run the broker or the cluster, keep the workers healthy, and run the triggerer for you. Your DAGs stay identical; you’ve just handed someone else the part of this post that involves getting paged. Still worth knowing what’s underneath: the trade-offs — throughput, isolation, latency, cost — don’t disappear because someone else operates them.
Final thoughts
The executor is the least glamorous choice in an Airflow deployment and one of the most consequential. It never shows up in your DAG code, which is the good news — you can start on LocalExecutor and move to Kubernetes years later without touching a task. But it decides whether your pipeline scales by adding a box, adding a worker, or adding a pod, and each of those is a different bet about where your pain will come from. The 3.x change worth internalizing is that this stopped being a single, all-or-nothing decision: you can run a list of executors and let each task pick, so the warm-worker jobs and the isolated-pod jobs finally coexist without a hybrid hack. Pick a default for the workload you have, reach for executor= on the exceptions, and let the clean separation between scheduler and executor mean you can change your mind later.
Comments