Reading the UI When a DAG Goes Red

Everything works until it doesn’t, and the first time a task turns red you’ll want to know exactly where to click. The Airflow 3.x UI is a React app that rewards a few minutes of orientation — it’s less a dashboard than a debugger for your pipelines, and the difference between a five-minute fix and a lost afternoon is knowing which panel holds the answer. This chapter walks the whole loop: find the failure, read it, reproduce it locally if you have to, rerun the slice you fixed — and, so you’re not the last to know, get Airflow to tell you instead of the other way around.

The panels, in the order you’ll use them

The DAG list is home: every DAG you have, its schedule, whether it’s paused, and the state of its recent runs as a row of colored dots. Green is success, red is failed, a soft amber is running or retrying. This is your triage screen — you scan for red and click in.

Clicking a DAG lands you on the grid view, and this is the one you’ll live in. It’s a matrix: runs along one axis, tasks along the other, each cell a colored square for that task in that run. A vertical stripe of green with one red square tells you exactly which task failed on which run, at a glance, without reading a single log. A whole red column means one task is failing every run — a systemic problem. A single red cell in a sea of green means one bad day’s data. The grid turns “something’s wrong” into “this task, this run” in one look.

The graph view answers a different question: not what failed but what’s downstream. It draws the DAG’s dependency shape, so when a task fails you can see instantly what got skipped or blocked behind it. Grid for when, graph for what depends on what.

And the logs panel: click any task instance and you get its stdout, stderr, and the full Python traceback. This is where the actual answer lives. Everything else just gets you here fast.

That’s the core three. But the 3.x UI has a handful of quieter panels that turn a hard debugging session into an easy one, and most people never open them because nobody pointed at them. Here’s the pointing.

The panels you’ll be glad you knew about

Open a task instance and you get a tab strip; open a DAG and you get a few more views hanging off it. Worth knowing, roughly in the order a real investigation reaches for them:

Rendered Template. This is the single most useful debugging panel in Airflow and the one newcomers miss longest. When you write bash_command="load {{ ds }}.csv", the string with {{ ds }} in it is the template — Airflow fills it in at run time. The Rendered Template tab shows you the result: the actual command, with the actual date substituted, for that specific run. Nine “why did it load the wrong file” bugs out of ten are visible here in two seconds — you templated {{ ds }} where you meant {{ ds_nodash }}, or a Variable came back empty and rendered a path with a hole in it. Read the rendered value, not the source; the source lies about what ran.
XCom. The tab that shows what a task returned — the small values tasks pass to each other (chapter 6). If transform is choking on fetch’s output, open fetch’s XCom tab and look at exactly what it handed downstream. Often the payload changed shape upstream and you can see it right there.
Task Instance Details. The full record for one run of one task: which operator, which queue and pool, its start and end times, the try number, its configured execution_timeout, and — in 3.x — which DAG version it ran against. When behavior doesn’t match the code you’re reading, this tab tells you whether you’re even looking at the same code the run used.
Code. The DAG’s source as Airflow parsed it, not as it sits in your editor. If your local file and this panel disagree, your change hasn’t been picked up yet — you’re debugging a version that isn’t deployed. Checking this before anything else has saved more afternoons than any traceback.
Gantt. A timeline of a single run: each task as a horizontal bar, positioned by when it started and how long it ran. This is your view for “the DAG is slow, not broken.” Long bars are hotspots; big gaps between bars are tasks waiting — for a pool slot, an upstream dependency, a sensor. Gantt is where “it takes three hours now” becomes “this one task takes three hours now.”
Task Duration. The same question across time: how long a given task took on each of the last N runs, plotted as a trend. A task that crept from thirty seconds to nine minutes over two weeks shows up as a rising line here long before it shows up as a failure. This is the panel that catches the slow rot.
Calendar. A month grid of run outcomes — green and red squares by date. Good for spotting patterns your eye misses in the grid: every Monday fails, or the failures started exactly on the 3rd.
Audit / Event Log. The answer to “who cleared this task?” and “did someone pause the DAG?” Every manual action — clears, mark-success, pause toggles, trigger-runs — is recorded with a user and a timestamp. When a run’s state doesn’t match what the schedule should have produced, a human usually reached in, and this is where you find out who and when.

You don’t need to memorize these. You need to know they exist, so that when you hit a wall the reflex is “there’s a panel for this” instead of “I’ll add a print statement and redeploy.”

The debugging loop

The loop is short and you’ll run it constantly. A task goes red. Open the grid, click the red cell, read its log, scroll to the traceback. Nine times out of ten the last few lines name the file, the line, and the exception — a missing column, a connection that timed out, a KeyError on a payload that changed shape. If the exception is about a value rather than a bug — a wrong filename, an empty config — flip to Rendered Template before you touch the code. Then you fix it.

Now the part people miss. You don’t rerun the whole DAG. You clear the failed task.

Clearing a task instance wipes its state and hands it back to the scheduler, which reruns it. By default it reruns the failed task and everything downstream of it, since those tasks consumed its output. So a fix to transform reruns transform and whatever depended on it, while the untouched upstream fetch and load are left exactly as they were. You re-execute the broken slice of the DAG, not the expensive parts that already succeeded. This is why the idempotency habit matters so much: clearing a task is running it again, and Airflow makes that one click.

The clear dialog gives you toggles for how far the blast radius reaches, and they’re worth understanding because the defaults aren’t always what you want:

Downstream (on by default) — the cleared task and everything after it. The normal case: you fixed transform, so transform and its dependents rerun.
Upstream — the cleared task and everything before it. Rare, but the move when you suspect the failure was actually caused by stale upstream output and you want to rebuild the whole chain leading in.
Only failed — within the selection, clear only the tasks that failed, leaving successful ones alone. Pair this with Downstream when a fan-out step had four branches and only one broke: clear the failed branch and its tail, not the three that worked.
Recursive — follow into TaskGroups and across into triggered DAGs, clearing their tasks too. For nested pipelines; you’ll know when you need it.
Past and Future — clear this same task across other runs, backward or forward in time. This is how you replay a bug fix across a week of already-completed runs: select the task, tick Past, choose how far back, and Airflow re-runs it for every historical run in range. Powerful and expensive — it can kick off a lot of work, so read the count before you confirm.

Sometimes the fix isn’t code. If a task failed because of a transient outside event you’ve since handled by hand, you can mark it success to let the run proceed; you can mark failed to stop a run you know is doomed. You can do this at the single-task level or, from the run itself, at the DAG-run level — mark an entire run success or failed in one action. When you re-trigger a run manually in 3.x, you can also pass a conf — a small JSON blob of parameters the run reads through {{ dag_run.conf }} — which is how you re-run “the same DAG, but for this one customer” or “with full_refresh: true” without editing code. Use mark-success and mark-failed sparingly, though: they’re you overriding Airflow’s record of reality, which is fine when reality genuinely changed and dangerous as a habit.

Reproduce it locally, before you clear anything

Clearing in the UI is a fine way to rerun a fixed task. It’s a terrible way to iterate on a fix — each attempt is a full scheduled round-trip, and you’re debugging through a web page. When a failure is fiddly, pull the single task down to your machine and run it in a tight loop. Three ways, smallest to largest.

airflow tasks test runs one task, right now, in your terminal, ignoring the scheduler and the database’s opinion of state entirely:

airflow tasks test bookshop_etl transform 2026-06-18

That’s <dag_id> <task_id> <logical_date>. It executes the task’s code with the real templated context for that date, streams the log straight to your terminal, and writes nothing to the metadata DB — no state change, no XCom persisted, safe to run fifty times. This is the fastest way to answer “does my fix work” without touching the deployed pipeline.

dag.test() runs the whole DAG in a single local process — no scheduler, no workers, no queue — which is the right tool when the bug is in how tasks hand off to each other, not in one task alone. Add a __main__ block to your DAG file.

The detail that trips people: with the @dag decorator, there is no variable called dag lying around. The decorated function returns the DAG when you call it — that bare bookshop_etl() at the bottom of the file, the one that looked like a no-op back in post 03, is what registers it. So you call .test() on the object the function hands back:

@dag(schedule="@daily", start_date=datetime(2026, 1, 1), catchup=False)
def bookshop_etl():
    ...

bookshop_etl()          # registers the DAG for the parser

if __name__ == "__main__":
    bookshop_etl().test()

Then just run the file: python dags/bookshop_etl.py. Airflow executes every task in dependency order in-process, prints each log inline, and you can drop a breakpoint() anywhere in your task code and actually step through it in a debugger — something you cannot do to a task running on a remote worker. For a newcomer, dag.test() is the single biggest upgrade to your debugging life: your DAG becomes an ordinary Python script you can run and step through.

astro dev pytest is for when the fix is worth a test that stays. The Astro CLI ships a pytest runner wired to your project — it runs whatever is in tests/, the directory astro dev init scaffolded back in chapter 2 and that you have, if we’re honest, ignored ever since:

astro dev pytest

Which raises the fair question: what actually goes in there? Two things, and the first one is worth more than people expect.

The import-integrity test. The most common Airflow failure isn’t a subtle logic bug — it’s a DAG file that doesn’t import. A typo, a missing package, a circular import. In production that shows up as an import error in the UI and a DAG that silently isn’t there. One test catches every instance of it, forever:

# tests/dags/test_dag_integrity.py
from airflow.models import DagBag


def test_no_import_errors():
    dag_bag = DagBag()
    assert not dag_bag.import_errors, f"DAG import failures: {dag_bag.import_errors}"


def test_every_dag_has_tags_and_retries():
    dag_bag = DagBag()
    for dag_id, dag in dag_bag.dags.items():
        assert dag.tags, f"{dag_id} has no tags — it'll be unfindable in the UI"
        assert dag.default_args.get("retries") is not None, f"{dag_id} sets no retries"

DagBag parses every file in dags/ exactly the way the dag-processor does, and collects the failures instead of raising. The first test is a smoke alarm — it fails the moment anyone commits a DAG that can’t be imported, which is precisely the bug that’s most embarrassing to find in production and cheapest to find in CI. The second is a policy test, and it’s the one people don’t think to write: it doesn’t check that a DAG is correct, it checks that it meets your team’s house rules. Every DAG has tags. Every DAG sets retries. You can extend that list as your standards harden, and the standards then enforce themselves on every pull request instead of living in a wiki nobody reads. (If you’ve seen DagBag(include_examples=False) in older code: that kwarg is gone in Airflow 3, and you don’t need it — a bare DagBag() reads your dags/ folder and nothing else.)

The unit test of a task’s logic. The other thing worth testing is the part of your code that isn’t really Airflow at all. A @task wraps an ordinary Python function, and the function is testable as a function — call .function to reach through the decorator:

# tests/dags/test_bookshop.py
from dags.bookshop import summarize_orders


def test_summarize_counts_only_completed():
    orders = [
        {"id": 1, "status": "completed"},
        {"id": 2, "status": "cancelled"},
        {"id": 3, "status": "completed"},
    ]
    assert summarize_orders.function(orders) == 2

No DagBag, no scheduler, no database — a millisecond. That’s the whole trick to testing Airflow: keep the logic in plain functions and the Airflow in thin wrappers, and the interesting half of your pipeline becomes as testable as any other Python. When a task is hard to test, it’s usually a sign the task is doing too much.

The progression is: reproduce with tasks test, iterate with dag.test() and a breakpoint, then lock the fix in with a pytest so the bug can’t quietly return. The Practice series goes much deeper — mocking the context, testing mapped and branching tasks, wiring this into CI — but the two tests above are the ones that pay for themselves in week one, and they belong in every project from the day it’s created.

When nothing runs at all

The failure mode with no red square is the confusing one: the scheduler seems stuck, nothing runs, the grid just sits there. Before you suspect the engine, check three things in order.

First, import errors — a DAG that raises on import can’t be scheduled, and Airflow surfaces these as a banner at the top of the UI; a typo or a bad import will silently sideline a DAG. Second, is the DAG paused? A paused DAG shows a toggle in the list and simply doesn’t schedule; a surprising number of “it’s broken” reports are a switch left off. Third, is there an eligible run to make — has the current data interval actually closed, given the time model from the scheduling chapter? A @daily DAG deployed an hour ago has nothing to run yet, and that’s correct, not stuck.

But there’s a fourth, sneakier version: a task that goes to queued and stays there, forever, never reaching running. That’s not a code bug — the task never got to run its code — it’s a capacity problem. Airflow queued the task, meaning it’s ready and waiting for a slot, and no slot ever opened. The usual culprits:

No free worker slots. Your executor has a fixed number of concurrent slots. If every slot is busy with other tasks, new ones queue behind them. On a small local or single-worker setup this happens the moment you fan out wider than the worker can handle.
An exhausted pool. Pools cap concurrency for a named resource — say, “only 3 tasks may hit the warehouse at once.” A task assigned to a full pool waits for a peer to finish and release its slot. If the pool is misconfigured to zero slots, tasks in it wait forever. Check the pool’s usage in the UI.
Executor saturation. Every task-level cap — the DAG’s max_active_tasks, the global parallelism setting — is a valve that can pinch. If the scheduler is willing to run but the executor has no headroom, tasks pile up in queued.

The tell is uniform: lots of gray, nothing green-lime, and it clears on its own the moment capacity frees up. If it doesn’t clear and the task sits queued while slots are visibly free, you’re likely looking at a zombie task — a task the scheduler believes is running on a worker that has actually died or lost its heartbeat. Airflow’s zombie detection notices the missing heartbeat within a couple of minutes, marks the task failed (so your retries kick in), and logs it as a zombie. If you see tasks flip to failed with a “zombie” note in the logs and no traceback of their own, that’s a worker or infrastructure problem, not your DAG — the code never got the chance to throw.

Getting told, instead of finding out

Everything so far assumes you’re already looking at the UI. In real life you want the reverse: the pipeline reaches out when something breaks, and you open the UI in response. This is the question every newcomer eventually asks — “Airflow used to have SLAs, so how do I get alerted?” — and the honest answer is that SLAs were removed in 3.x. The old sla=timedelta(...) argument is gone; set it and it’s silently ignored. Alerting is now done with callbacks, and it’s better for it, because callbacks fire on the actual event rather than on a timer’s guess.

Every task and DAG accepts callback hooks — functions Airflow calls, with the run’s context, when a specific thing happens:

on_failure_callback — the task failed for good (retries exhausted). This is your page.
on_retry_callback — the task failed but will try again. Lower-stakes signal: nice for a channel you skim, not one that wakes someone.
on_success_callback — the task succeeded. Usually noise, but occasionally you want a “the nightly finished” ping.

The quickest wins don’t even need a function. On any operator, email_on_failure=True and email_on_retry=True send mail to the task’s email addresses through your configured mail connection — zero custom code, good enough for a solo project. For a team you’ll want Slack or PagerDuty, and the provider packages ship notifiers — ready-made callback objects — so you don’t write the HTTP call yourself:

from datetime import timedelta
from airflow.sdk import dag, task
from airflow.providers.slack.notifications.slack import SlackNotifier

nightly_failed = SlackNotifier(
    slack_conn_id="slack_alerts",
    text="Bookshop nightly failed — {{ ti.dag_id }}.{{ ti.task_id }} "
         "on {{ ds }}. Logs: {{ ti.log_url }}",
    channel="#data-alerts",
)

@dag(
    schedule="@daily",
    catchup=False,
    default_args={
        "retries": 2,
        "retry_delay": timedelta(minutes=5),
        "on_failure_callback": nightly_failed,
    },
)
def bookshop_etl():

    @task
    def transform():
        ...

    transform()

bookshop_etl()

Setting the callback in default_args wires every task in the DAG at once, so any red square posts to #data-alerts with a link straight to its own log — the {{ ti.log_url }} template renders the exact URL of the failed task instance, so the person reading the alert is one click from the traceback. That link is the whole point: an alert that only says “something failed” makes someone go hunting; an alert that deep-links the log makes the fix start immediately.

Callbacks cover “it broke.” The other half — “it ran, but far too long” — is what SLAs used to (badly) approximate, and 3.x replaces them with Deadline Alerts. A DeadlineAlert fires a callback if a run hasn’t finished within a window you anchor to a reference point:

from datetime import timedelta
from airflow.sdk import dag, DeadlineAlert, DeadlineReference, AsyncCallback
from airflow.providers.slack.notifications.slack import SlackNotifier


# must be a module-level *awaitable* — see the note below
async def notify_slack_late(context):
    SlackNotifier(
        slack_conn_id="slack_alerts",
        text="Bookshop nightly is 90 min past its logical date and still running.",
        channel="#data-alerts",
    ).notify(context)

@dag(
    schedule="@daily",
    catchup=False,
    deadline=DeadlineAlert(
        reference=DeadlineReference.DAGRUN_LOGICAL_DATE,
        interval=timedelta(minutes=90),
        callback=AsyncCallback(notify_slack_late),
    ),
)
def bookshop_etl():
    ...

reference sets where the clock starts — here the run’s logical date — interval is the budget, and the scheduler fires the callback if the run hasn’t finished by then. Unlike the old SLA, which was tied to a task starting, a deadline anchors to a point you choose, so “done within ninety minutes of the logical date” is finally expressible. This is the modern “this run took too long” mechanism; the Practice series goes further into routing retries and failures to different destinations.

Two related valves are worth naming here because they show up right next to alerting in the UI. execution_timeout (per task) and dagrun_timeout (per DAG) don’t notify — they kill. Set execution_timeout=timedelta(minutes=30) on a task and Airflow forcibly fails it if it runs longer, which then triggers your on_failure_callback. A deadline alert tells you a run is slow; a timeout stops a run that’s hung. You usually want both: a timeout so nothing runs forever, and a deadline (or callback) so you hear about it.

Where the logs actually live

Every task instance streams its stdout and stderr to a log, and by default those files live on disk under the scheduler/worker’s log directory, keyed by DAG, run, task, and attempt number. That last key matters: a task with retries=2 can produce three logs, and the UI gives you a per-attempt selector — a little dropdown on the log panel — so you can read attempt 1’s failure, attempt 2’s failure, and attempt 3’s success separately. When a flaky task “eventually worked,” the interesting story is in the earlier attempts, and the selector is how you get to them. If your own code uses Python’s logging, those lines land in the same log at their level (INFO, WARNING, ERROR), and Airflow’s log verbosity is configurable — turn it up when you’re chasing something, down when it’s noisy.

Local disk is fine on your laptop and wrong in production, because workers are ephemeral — the box that ran last night’s task may not exist today, and its logs with it. The fix is remote logging: point Airflow at an S3 bucket or GCS path and it ships every task log there as the task finishes, then serves it back into the same UI panel. You configure it once (a remote base folder plus a connection), and nothing about how you read logs changes — the panel just now reads from durable storage. We set this up properly in the Practice series; for now, know that “the log is gone because the worker died” is a solved problem and not something to design around.

A field guide to the colors

The grid speaks in color, and knowing the vocabulary means you read a run’s state without clicking. The ones you’ll actually meet:

State	Color	What it means
success	green	Ran and finished cleanly.
running	bright lime	Executing right now.
failed	red	Raised, and retries (if any) are exhausted. This is the one you click.
up_for_retry	amber/gold	Failed, but will try again after `retry_delay`. Not red yet.
up_for_reschedule	turquoise	A sensor in `reschedule` mode, sleeping until its next poke. Waiting, not stuck.
queued	gray	Ready to run, waiting for a free slot. Lingers here → capacity problem.
scheduled	tan	The scheduler has decided it should run; about to be queued.
deferred	purple	Handed off to the triggerer, freeing its slot while it waits (the async story, covered in Practice).
upstream_failed	orange	Never ran because something it depended on failed. Chase the upstream red square, not this one.
skipped	pink	Deliberately not run — a branch not taken, a trigger rule that said no.
none / no status	pale	The scheduler hasn’t gotten to it yet.

The two that trip newcomers up are upstream_failed (orange) and skipped (pink): both mean “this task didn’t run,” but orange is a problem you should follow backward and pink is by design. Learn to tell them apart at a glance and half of “why didn’t this run” answers itself before you open a single log.

Final thoughts

The UI stops being intimidating the moment you treat it as a debugger rather than a status board. Grid to find the failed cell, log to find the traceback, Rendered Template when the value’s wrong, clear to rerun the slice you fixed — and when a failure is fiddly, pull it down to dag.test() and step through it like ordinary Python. Wire one on_failure_callback so the pipeline pages you instead of the other way around, and you’ve closed the loop: it tells you it broke, links you to the log, and you fix and clear in three clicks.

That’s the whole muscle, and it becomes reflex within a week. The engineers who look fast at Airflow aren’t reading logs faster than you; they just click the red square first instead of scrolling everything, they check “paused” before they blame the scheduler, and they check the per-attempt selector before they swear the task never logged anything.

That closes the loop on the core mechanics: you can now write a DAG, schedule it, wire it to the outside world, and read the UI when it breaks — which is enough to run real pipelines. The remaining Foundations chapters fill in the control flow and the plumbing that every real pipeline eventually needs: branching and trigger rules, fanning out over a list, templating, and where connections and secrets actually come from.

Next: Branching and Trigger Rules