Rolling Your Own Operators and Hooks
When you've copy-pasted the same @task one time too many, it's time to give your integration a name — a custom operator, backed by a hook that knows how to talk to your system.
Airflow in Practice
Custom operators, sensors, dynamic mapping, assets, executors, and shipping to production — Airflow 3 beyond the basics.
-
-
Sensors and Deferrable Operators: Waiting Without Wasting
Half of data engineering is waiting for a file to land. Do it wrong and a sensor pins a worker for hours; do it right and the task steps aside until the world is ready.
-
Dynamic Task Mapping and TaskGroups: Fan Out, Stay Tidy
You don't know how many regional files today's run will bring until the run starts — so let Airflow build the tasks it needs at runtime, then keep the graph readable while it does.
-
Assets: Scheduling on Data, Not the Clock
Airflow 3's Assets let a DAG run the moment fresh data lands, instead of guessing the upstream lag with cron.
-
Executors: Local, Celery, and Kubernetes
The scheduler decides what's ready to run; the executor decides how and where it actually runs. Pick the wrong one and you feel it.
-
Containerized Tasks: Run Anything, Blame Nothing
Your worker's Python is not your task's Python. DockerOperator and KubernetesPodOperator let a task bring its own universe — and hand you a debugging story that now spans two systems.
-
Delegating Compute: Airflow Is the Conductor, Not the Orchestra
A task that pulls a million rows into the worker's memory is a bug. Push the work to dbt, the warehouse, and Spark — and keep the scheduling, the retries, and the record of what happened.
-
Production Patterns: Idempotent, Observable, Alerting
The three habits that separate a DAG you trust in production from one you babysit: safe re-runs, a callback that pages you, and a source database you don't melt.
-
Testing DAGs and Wiring Up CI
Most broken DAGs never make it past a five-line import test — here's how to catch them before merge instead of at 3am.
-
Shipping It: Astronomer, MWAA, and Cloud Composer
Your DAGs run on your laptop. Getting them running somewhere that pages a rotation instead of you is the last mile — here are the three managed roads and what each one costs.
-
Branching, Setup/Teardown, and Params
Conditional production DAGs need explicit joins, resource lifecycle tasks, and typed params instead of ad hoc conf dictionaries.
-
Custom XCom Backends
Why large XCom payloads hurt the metadata database, and how object-storage-backed XComs keep Airflow passing references instead of blobs.
-
Secrets Backends in Depth
A production treatment of Vault, AWS, and GCP secrets patterns, lookup order, prefixes, rotation, and why UI-stored secrets rarely scale.
-
REST API, External Triggers, and Dynamic DAGs
How other systems start DAG runs, how conf and params meet, and how to generate many DAGs without making parsing expensive.
-
Plugins, Listeners, and Notifiers
The extension points for UI additions, lifecycle hooks, and provider-backed notifications in modern Airflow.
-
Performance, Scheduler Tuning, and Monitoring
The knobs that decide parse speed, task throughput, queueing behavior, and whether you notice the scheduler falling behind.
-
Kubernetes and Multi-Executor Patterns
Per-task pods, pod overrides, templates, CeleryKubernetes tradeoffs, and the triggerer role that cuts across executors.
-
Who Gets to Press the Buttons
Auth managers, FAB roles, DAG-level access control, JWT, and the honest limits of Airflow's multi-tenancy — the chapter that decides who can clear a task at 2am.