Airflow: When cron Stops Being Enough
How a folder of cron jobs turns into a silent failure, and what an orchestrator does about it.
Airflow from Scratch
Orchestration for newcomers — why it exists, running Airflow locally, and building your first DAGs with the TaskFlow API on Airflow 3.
-
-
Your Airflow, Running in One Command
Scaffold a real Airflow project and bring the whole stack up locally with the Astro CLI.
-
Your First DAG: Tasks That Know Their Order
Write a two-task bookshop pipeline in Python and watch the scheduler run it in order.
-
Operators: The Verbs of a DAG
A DAG is a sentence about your data. Operators are the verbs — prebuilt task templates for running a command, calling an API, or executing SQL, plus the retry and timeout knobs that make them survive contact with production.
-
The TaskFlow API: DAGs That Look Like Python
Decorate a function with @task, call it, and Airflow builds the graph for you. The TaskFlow API turns a DAG into code you can almost read aloud.
-
XComs: How Tasks Pass Notes
When one task needs to tell the next one something small — a row count, a file path — it uses an XCom. Here's how they work, and the one way people misuse them.
-
Schedules, Catchup, and Backfills Without the Fear
A daily DAG that runs at midnight is processing yesterday, not today. Once that clicks, schedules, catchup, backfills, timezones, and Assets stop being scary.
-
Connections and Hooks: Talking to the Outside World
Your DAG needs to reach a database, an API, a bucket — without a password ever appearing in your code. Connections and hooks are how.
-
Reading the UI When a DAG Goes Red
A task failed. Somewhere in the UI is the traceback that tells you why — and the button that reruns just that task. Here's the whole loop, plus how to get paged before you're staring at a red square.
-
Branching and Trigger Rules
How Airflow represents conditional paths, skipped tasks, joins, and the trigger rules that keep branches from breaking the graph.
-
Dynamic Task Mapping and TaskGroups
Fan out over runtime data with expand, partial, and expand_kwargs, then keep the graph readable with TaskGroups.
-
Templating, Macros, and Rendered Fields
The Jinja surface that powers templated SQL, shell scripts, dates, params, connections, variables, and the Rendered Template view.
-
Connections, Variables, and Secrets Backends
How Airflow stores external configuration, when to use Variables, how hooks read Connections, and how secrets backends change the lookup path.