Performance, Scheduler Tuning, and Monitoring

The knobs that decide parse speed, task throughput, queueing behavior, and whether you notice the scheduler falling behind.

The first thing to internalize about Airflow performance is that most of it happens before a single task runs. By the time a bookshop extract is actually pulling orders, the interesting decisions — how fast the DAG got noticed, how long the scheduler took to mark it ready, whether there was a free slot to run it in — are already made. Those decisions live in three places: the parser that turns your .py files into DAG objects, the scheduler loop that decides what’s runnable, and the stack of concurrency ceilings that decides how much runs at once. When someone says “Airflow feels slow,” it’s almost always one of those three, and they’re almost always looking at the fourth thing — the worker — where the problem isn’t.

This chapter walks the three in order, then wires up the metrics that tell you which one is actually hurting, and ends with a runbook for the day the whole thing feels sluggish and you don’t yet know why.

Parsing is the hidden cost

Airflow doesn’t parse your DAG files once. It parses them over and over, forever, because a DAG file is live Python and the only way to know whether you changed it is to re-execute it. Every file in the DAGs folder gets imported on a loop, and the DAG objects it produces get serialized into the metadata database for the scheduler to read. This is the single most misunderstood cost in an Airflow deployment, and it’s the reason for the oldest rule in the book: do no real work at the top level of a DAG file.

Top-level code is anything that runs at import time — module scope, outside any task function. The parser executes it on every single parse, of every file, several times a minute. So this is a slow-motion disaster:

# module scope — runs on EVERY parse, several times a minute
import pandas as pd
from my_company.warehouse import get_connection

conn = get_connection()                      # opens a DB connection at parse time
active_books = pd.read_sql("SELECT ...", conn)  # queries the warehouse at parse time

@dag(schedule="@daily", catchup=False)
def bookshop_report():
    @task
    def summarize():
        ...

Every parse now opens a warehouse connection and runs a query. Multiply by “every file, every 30 seconds, across a dozen scheduler parse cycles” and you’ve built a denial-of-service loop against your own database — and made every parse take as long as that query, which stalls the whole DAGs folder behind it. The fix is to push the work inside a task, where it runs once per DAG run instead of on every parse:

@dag(schedule="@daily", catchup=False)
def bookshop_report():
    @task
    def summarize():
        import pandas as pd                    # import inside the task
        from my_company.warehouse import get_connection
        conn = get_connection()                # connection at RUN time, once
        active_books = pd.read_sql("SELECT ...", conn)
        ...

The rule generalizes past database calls: no API requests, no reading large files, no expensive imports, no dynamic DAG generation that loops over a remote list at module scope. Heavy imports are the sneaky one — import pandas at the top of forty DAG files makes every parse of all forty pay for pandas’ import cost, even though only one task uses it. Push non-trivial imports into the tasks that need them. Whatever must run at parse time — the DAG structure itself, the dependency wiring — should be cheap, deterministic, and free of I/O.

The knobs that govern parsing

Assuming your files are clean, four settings control how aggressively Airflow re-parses:

AIRFLOW__DAG_PROCESSOR__MIN_FILE_PROCESS_INTERVAL=30
AIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL=300
AIRFLOW__DAG_PROCESSOR__PARSING_PROCESSES=2
AIRFLOW__DAG_PROCESSOR__DAG_FILE_PROCESSOR_TIMEOUT=50

refresh_interval is the one that used to be called [scheduler] dag_dir_list_interval — it was renamed in 3.0 when parsing moved into the standalone dag processor, and the old name lingers in every 2.x tutorial you’ll find.

min_file_process_interval is the floor on how often a given file is re-parsed — 30 seconds by default. Lower it and edits show up in the UI faster, at the cost of more parse load; raise it (60, 120) on a big, stable DAGs folder and you cut parsing CPU noticeably, trading away only how quickly a code change is picked up. dag_dir_list_interval is how often Airflow re-scans the directory for new or deleted files (300 seconds default) — this is about noticing that a file appeared, not about re-parsing existing ones. parsing_processes is how many files the processor chews through in parallel; on a host with spare cores and hundreds of DAG files, bumping it from 2 to 4 or 8 shortens the time to get through the whole folder.

The standalone dag-processor

In Airflow 3, parsing runs in its own component: the dag-processor. This is a real architectural change from 2.x, where the scheduler could parse files in-process. Now the dag-processor imports your files, and the scheduler only reads the serialized DAGs out of the database — it never executes your DAG code at all. That’s a security and stability win (a DAG file that hangs or leaks memory can’t take the scheduler down with it), and it means parse load and schedule load can be reasoned about, and scaled, separately.

Locally, astro dev start runs a dag-processor for you. In a hand-rolled deployment it’s a process you have to start:

airflow dag-processor

Forget it and DAGs simply never appear — the scheduler is happily reading an empty serialized table because nothing is filling it. If your team is coming from 2.x, this is the most common “my new DAG isn’t showing up” cause: there’s no dag-processor running.

Measuring parse time, and .airflowignore

You don’t have to guess which file is slow. The dag-processor writes per-file timing, and the CLI surfaces it:

airflow dags report

This lists every file with its parse duration and the number of DAGs it produced. A healthy file parses in tens of milliseconds; anything measured in seconds has real work happening at module scope, and airflow dags report will point straight at it. The aggregate — total time to parse the whole folder — is also exported as a metric (dag_processing.total_parse_time, which we’ll wire up below); watch that number and you’ll catch a folder sliding from “parses in 4 seconds” to “parses in 40” before it becomes an outage.

The cheapest parse is the one that never happens. An .airflowignore file in the DAGs folder tells the processor which paths to skip entirely — the same idea as .gitignore. Utility modules, test fixtures, a plugins/ subtree, generated SQL — none of it defines DAGs, so none of it should be parsed:

# dags/.airflowignore
utils/
tests/
.*_helpers\.py

In Airflow 3 the patterns are globs by default ([core] dag_ignore_file_syntax = glob); 2.x treated them as regexes, so an inherited .airflowignore full of .* patterns will quietly stop matching. Keeping non-DAG Python out of the folder — or ignoring it explicitly — means the processor spends its cycles on files that actually produce DAGs, which is the whole point.

The concurrency ceilings

Here’s the scenario that sends people to the forums: a task sits in the queued state, not running, and the workers look idle. Nothing is failing. It’s just stuck. The reason is that “can this task run?” is gated by a stack of independent ceilings, and the task runs only when it’s under all of them at once. Miss one and the task waits, silently, with no error to grep for. Let’s walk the stack from the top.

[core] parallelism is the global ceiling — the maximum number of task instances running across the entire deployment, regardless of DAG, pool, or worker. Default is 32.

AIRFLOW__CORE__PARALLELISM=32

If 32 tasks are already running anywhere, the 33rd waits, full stop. This is the master valve, and on a busy deployment it’s the one people forget they set low years ago.

max_active_tasks_per_dag caps concurrent tasks within a single DAG run’s worth of parallelism — how many tasks of one DAG run at once (default 16). It’s set globally and overridable per DAG:

@dag(schedule="@daily", catchup=False, max_active_tasks=8)
def bookshop_nightly():
    ...

A DAG that fans out into 200 mapped tasks won’t run 200 at once; it runs max_active_tasks at a time. Good for being polite to a downstream system, and a trap if you set it low and wonder why a wide DAG crawls.

max_active_runs_per_dag caps how many runs of the same DAG are active at once (default 16, but commonly set to 1). This is the one that bites during backfills:

@dag(schedule="@daily", catchup=False, max_active_runs=1)
def bookshop_nightly():
    ...

With max_active_runs=1, a catchup or backfill of thirty days runs the days one at a time — day two doesn’t start until day one finishes. That’s usually what you want for a source you’re protecting, but if you’re staring at a backfill that seems to be executing serially when you expected parallelism, this is why. It’s a correctness knob as much as a speed one.

Pool slots cap concurrency by resource rather than by DAG. You define a pool with N slots and assign tasks to it; no more than N of them run at once, across every DAG:

@task(pool="source_db", pool_slots=1)
def extract_orders():
    ...

The source_db pool with 4 slots means at most four tasks touching that database run concurrently, whether they’re in one DAG or ten. Pools are how you protect a shared resource that DAG-level limits can’t see. (A task can also claim more than one slot with pool_slots if it’s especially heavy.)

worker_concurrency (Celery only) caps how many tasks a single worker process runs at once (default 16). Total Celery capacity is worker_concurrency × number of workers — and that product is itself capped by parallelism.

AIRFLOW__CELERY__WORKER_CONCURRENCY=16

Worked: “why is my task stuck in queued?”

Now the diagnosis, in the order I actually check it. A bookshop extract is queued and not running:

  1. Global parallelism. Count running task instances across the whole deployment. At or above parallelism? That’s it — the global valve is shut, and the task waits for any other task anywhere to finish. Symptom: lots of tasks queued across many DAGs at once.
  2. The pool. Is the task in a pool? Check the pool’s used vs. total slots in the UI (Admin → Pools). Zero open slots means the task waits for a pool-mate to finish, even if the rest of the deployment is idle. Symptom: only tasks in that pool are stuck; everything else runs.
  3. max_active_runs on the DAG. Is another run of the same DAG already active and holding the limit? Then this run hasn’t even started scheduling its tasks. Symptom: the DAG run is queued/waiting, not just the task.
  4. max_active_tasks within the run. Are max_active_tasks sibling tasks already running in this run? The rest queue behind them. Symptom: some tasks of this DAG run, a specific number, and the rest wait.
  5. Executor capacity. Only now do you look at workers. On Celery, are all worker_concurrency slots on every worker full? On Kubernetes, is the cluster refusing to schedule more pods (quota, no nodes)? Symptom: open_slots at zero on the executor metric.

The discipline is to walk it top-down. Ninety percent of “stuck in queued” is one of the first four — a ceiling you set and forgot — not the workers you’re tempted to blame first. The metrics section below gives you a number for each of these so you’re not counting rows by hand.

Tuning the scheduler loop

The scheduler runs a loop: examine the serialized DAGs, create any DAG runs that are due, find task instances whose dependencies are met, and hand the runnable ones to the executor. Do that as fast as possible, sleep briefly, repeat. A few knobs shape it.

AIRFLOW__SCHEDULER__MAX_DAGRUNS_TO_CREATE_PER_LOOP=10
AIRFLOW__SCHEDULER__SCHEDULER_IDLE_SLEEP_TIME=1

max_dagruns_to_create_per_loop caps how many new DAG runs the scheduler creates in one pass (default 10). It exists to keep one burst of due DAGs — say, the fifty things scheduled at midnight — from monopolizing a loop iteration and starving the scheduling of already-running tasks. If you have a lot of DAGs firing at the same instant and they’re slow to start, raising this lets the scheduler create runs faster, at the cost of longer, heavier loop iterations. scheduler_idle_sleep_time (formerly scheduler_heartbeat_sec-adjacent tuning) is how long the scheduler naps when a loop found nothing to do — lower it toward zero for snappier response on a quiet deployment, though the default is already fine for most.

The more important lever here isn’t a config value, it’s your DAG shape. The scheduler’s per-loop cost scales with the number of DAGs and task instances it has to evaluate. Many small DAGs spread that work out — each is cheap to evaluate, cheap to parse, cheap to reason about, and one failing doesn’t touch the others; the cost is more scheduling overhead in aggregate and a busier metadata database. Few large DAGs concentrate it — a single 500-task DAG is one parse and one dependency graph, but that graph is expensive to walk every loop, one bad task can wedge the whole run, and max_active_tasks becomes a bottleneck you feel. There’s no universal answer, but the failure mode to avoid is the pathological large one: a DAG with thousands of dynamically mapped tasks that the scheduler has to evaluate on every loop will slow scheduling for everything else on the deployment, because they share the loop. If one DAG is dragging the scheduler down, splitting it — or capping its fan-out — helps the whole system, not just that DAG.

The metrics catalog

You cannot tune what you cannot see, and the Airflow UI is a poor performance dashboard — it shows you individual runs, not trends. The trends live in metrics, which Airflow emits in two ways: the older StatsD protocol, and, in 3.x, OpenTelemetry (OTel) for both metrics and traces. OTel is the direction of travel — it puts Airflow’s telemetry in the same standard your other services already speak — but StatsD is still fully supported and slightly simpler to stand up, so I’ll show both.

The metrics worth watching, and what each one tells you:

  • scheduler.heartbeat — the scheduler emits this every loop. If it stops or slows, the scheduler is wedged or dying, and nothing will schedule. This is your number-one liveness signal; alert on its absence.
  • dag_processing.total_parse_time — how long it takes the dag-processor to get through the whole DAGs folder. Rising steadily means a file (or the folder as a whole) is getting expensive to parse — the top-level-code problem, caught early. This is the metric behind the parsing section above.
  • dag_processing.import_errors — count of files that failed to import. Nonzero means a DAG is broken and invisible.
  • executor open_slots and queued_tasks — the executor reports how much capacity is free and how much work is waiting. open_slots at zero with queued_tasks climbing is textbook slot exhaustion: the executor is full and a backlog is forming. This is the “stuck in queued” story as a graph.
  • dagrun.schedule_delay.<dag_id> — the gap between when a run was supposed to start and when it actually did. Growing delay means the scheduler is falling behind its own schedule.
  • task duration (dag.<dag_id>.<task_id>.duration) and ti.finish / ti.start counters — per-task timing and throughput, for spotting the task that’s quietly getting slower.

Wiring it up: StatsD → Prometheus → Grafana

The classic pipeline points Airflow’s StatsD output at a statsd-exporter, which Prometheus scrapes, and Grafana graphs:

AIRFLOW__METRICS__STATSD_ON=True
AIRFLOW__METRICS__STATSD_HOST=statsd-exporter
AIRFLOW__METRICS__STATSD_PORT=9125
AIRFLOW__METRICS__STATSD_PREFIX=airflow

Airflow pushes metrics as UDP packets to the statsd-exporter, which translates them into the Prometheus format on a /metrics endpoint; Prometheus scrapes that, and Grafana queries Prometheus. The one subtlety is that StatsD metric names contain the dag_id and task_id as dotted path segments (airflow.dag.bookshop_nightly.extract.duration), so to get clean Prometheus labels you give the exporter a mapping config that turns those segments into dag_id="bookshop_nightly" labels. It’s boilerplate, but it’s the difference between one queryable airflow_task_duration{dag_id=...} series and thousands of unlabeled metric names.

Wiring it up: OpenTelemetry

The 3.x-native path skips the StatsD translation and emits OTel directly to a collector:

AIRFLOW__METRICS__OTEL_ON=True
AIRFLOW__METRICS__OTEL_HOST=otel-collector
AIRFLOW__METRICS__OTEL_PORT=4318
AIRFLOW__METRICS__OTEL_INTERVAL_MILLISECONDS=30000

Now Airflow exports OTLP to an OpenTelemetry Collector, which can fan the data out to Prometheus, to a managed observability backend, or anywhere else — the collector is the routing layer, so you configure exporters there rather than reconfiguring Airflow. Metric names arrive with proper attributes rather than dotted paths, which is the main ergonomic win over StatsD.

Traces, not just metrics

The reason to prefer OTel isn’t the metrics — it’s the traces. With tracing enabled Airflow emits spans for a DAG run: the run is the root span, and scheduling, queueing, and each task’s execution hang off it as child spans.

AIRFLOW__TRACES__OTEL_ON=True
AIRFLOW__TRACES__OTEL_HOST=otel-collector
AIRFLOW__TRACES__OTEL_PORT=4318

Open one bookshop_nightly run in a trace viewer (Jaeger, Tempo, Grafana) and you see, on a single timeline, exactly where the wall-clock time went: how long the run sat queued before a slot opened, how long scheduling took, and which task dominated the runtime. That “queued” gap rendered as a visible span is the single most useful thing for diagnosing the concurrency-ceiling problems above — you stop guessing whether a run was slow because of the work or slow because it was waiting for a slot, because the waiting is right there as a bar on the chart.

The deployment feels slow — where do I look

When the whole thing feels sluggish and you don’t yet know why, resist jumping to “add more workers.” Walk it in the order the work actually flows: parse → schedule → run.

  1. Is the scheduler alive and keeping pace? Check scheduler.heartbeat is steady and dagrun.schedule_delay isn’t climbing. A stalled or lagging heartbeat means the scheduler itself is the bottleneck — often because one enormous DAG is dragging the loop (see the shape discussion above), or the metadata database is slow. Nothing downstream matters until the scheduler is healthy.

  2. Is parsing keeping up? Look at dag_processing.total_parse_time and run airflow dags report. If total parse time is high or a single file parses in seconds, you’ve got top-level work — a query, an import, an API call at module scope. Fix that file and everything speeds up, because a slow parse delays every DAG in the folder from being noticed. Check import_errors too, in case the “slow” DAG is actually a broken one that never appears.

  3. Are tasks starting promptly once ready? Now look at the executor: open_slots and queued_tasks. If slots are at zero and the queue is growing, you’re slot-exhausted — but which ceiling? Walk the stack from the concurrency section: global parallelism, then the relevant pool, then max_active_runs, then max_active_tasks, then actual worker/pod capacity. The metric tells you there’s a backlog; the ceiling walk tells you which valve is shut. Only if you reach the bottom of that list — genuinely full workers, nothing else constraining — is “add capacity” the right answer.

The trap at every step is skipping to the end. The worker fleet is the most visible, most tangible part of the system, so it’s where the eye goes — but a slow deployment is far more often a query running on every parse, a scheduler loop bogged down by one giant DAG, or a parallelism value someone set to 16 during testing and forgot. Walk parse → schedule → run, read the metric at each stage, and you’ll land on the real ceiling instead of paying for workers that then sit idle behind it.

Final thoughts

Almost none of this is about making tasks run faster — it’s about removing the things that keep them from running at all. The parser re-executing a warehouse query every thirty seconds, the scheduler loop grinding through a ten-thousand-task DAG, the parallelism valve someone throttled during a test and left throttled: these are the costs that don’t show up in any single task’s logs, which is exactly why they’re worth instrumenting. Get StatsD or OTel flowing into a dashboard before you need it, so that when the deployment feels slow you’re reading total_parse_time and queued_tasks instead of guessing. And keep the oldest rule in front of everything: do no real work at the top of a DAG file. Most Airflow performance problems are just that rule, violated, at scale — and the fix is deletion long before it’s a bigger box.

Next: Kubernetes and Multi-Executor Patterns

Comments