Shipping It: Astronomer, MWAA, and Cloud Composer

Your DAGs run on your laptop. Getting them running somewhere that pages a rotation instead of you is the last mile — here are the three managed roads and what each one costs.

Everything so far has run inside astro dev start on your machine. That’s a real Airflow — scheduler, api-server, workers, the React UI at localhost:8080 — but it evaporates when you close the laptop, and it’s serving exactly one user. Production means something that stays up, survives a node dying, keeps its metadata database backed up, and doesn’t make you the on-call for the infrastructure underneath your pipelines. You can run all of that yourself. Most teams shouldn’t. The interesting decision isn’t self-host versus managed — it’s which managed service, and what each one asks in return.

The three managed roads

Astronomer is the shortest path from this series’ setup, because this series’ setup is Astronomer’s local tooling. The project astro dev init scaffolded — the Dockerfile, requirements.txt, dags/, packages.txt — is the exact shape Astronomer deploys. One command ships it:

astro deploy

That builds the image, pushes it, and rolls it out to a Deployment you provisioned in their control plane. You configure the environment (Airflow version, worker resources, environment variables, secrets backend) through their UI or API and never touch a Kubernetes manifest. It’s a paid product on top of open-source Airflow, and the trade is money for not owning the platform. If you got this far with the Astro CLI, this is the road of least resistance.

AWS MWAA (Managed Workflows for Apache Airflow) is the AWS-native option. You point an environment at an S3 bucket holding your dags/ folder and a requirements.txt, pick an environment class, and AWS runs the scheduler and workers inside your VPC. It integrates cleanly with IAM, CloudWatch, and Secrets Manager, which is the whole appeal — if your data already lives in AWS, MWAA keeps the blast radius and the bill in one account. The cost is a heavier deploy loop (sync to S3, wait for the environment to pick it up) and less say over the exact Airflow build than Astronomer gives you.

GCP Cloud Composer is the same bargain on Google’s side. Composer runs Airflow on GKE, wired into Cloud Storage for DAGs, Cloud Logging for logs, and Secret Manager for secrets. If your warehouse is BigQuery and your stack is GCP, Composer is the native fit and the operators for Google services are first-class. Same shape of trade as MWAA: tight cloud integration, in exchange for a deploy that goes through a storage bucket and a platform you tune less directly.

Self-hosting — Airflow on your own Kubernetes via the official Helm chart — is real and plenty of teams do it. Just be honest that you’re signing up to run the metadata database, the scheduler’s availability, upgrades, and the 3am page when a worker node dies. That’s a platform team’s job, not a side quest. We’ll walk that road first, because seeing what the managed services do for you is the fastest way to understand what you’re paying them for.

Self-hosting on the official Helm chart

The community maintains an official chart at apache-airflow/airflow. It stands up every component we’ve talked about — scheduler, api-server, workers, triggerer — plus the plumbing that keeps them alive. You drive it with a values.yaml, and reading a production one top to bottom is the best map of what “running Airflow” actually entails.

helm repo add apache-airflow https://airflow.apache.org
helm repo update
helm upgrade --install airflow apache-airflow/airflow \
  --namespace airflow --create-namespace \
  --version 1.18.0 \
  -f values.yaml

The first decision is the executor, and it flows through the rest of the file:

# values.yaml
executor: CeleryExecutor          # or KubernetesExecutor — or both (see below)

airflowVersion: "3.3.0"
defaultAirflowRepository: your-registry.example.com/bookshop-airflow
defaultAirflowTag: "2026.06.01-abc1234"   # your baked image tag, not "latest"

With CeleryExecutor you run a fixed-ish pool of workers and a broker (Redis); with KubernetesExecutor every task is its own pod and there’s no standing worker fleet. If you want both behaviors, do not reach for the old CeleryKubernetesExecutor hybrid — Airflow 3 supersedes it with multiple executors per deployment (chapter 17): you configure a list, and route individual tasks with executor="KubernetesExecutor". Same outcome, no special hybrid class. For the bookshop I’d run Celery with KEDA autoscaling as the default — it’s cheaper for a stream of short tasks than paying pod-startup latency on each one — and send only the fat, isolation-hungry tasks to Kubernetes.

Workers, KEDA, and the broker

workers:
  replicas: 2
  resources:
    requests: { cpu: "500m", memory: "1Gi" }
    limits:   { cpu: "2",    memory: "4Gi" }
  keda:
    enabled: true
    minReplicaCount: 1
    maxReplicaCount: 10
    # KEDA queries the metadata DB for queued+running tasks and scales workers to match
    pollingInterval: 5
  persistence:
    enabled: false     # workers are stateless when logs go to S3/GCS (below)

redis:
  enabled: true        # the Celery broker; disable and point brokerUrl at managed ElastiCache if you prefer

KEDA is the piece that turns a static worker pool into something that tracks load: it reads the count of queued and running tasks straight from the metadata database and scales the worker Deployment between minReplicaCount and maxReplicaCount. Without it you’re sizing for peak and paying for peak at 3am. Note persistence.enabled: false on the workers — that’s only safe once task logs are shipped off-box to object storage, which you configure under config.logging with a remote_base_log_folder like s3://bookshop-airflow-logs/. Otherwise a scaled-down worker takes its logs with it.

The metadata database and PgBouncer

Airflow’s metadata Postgres is the single most important thing to not run casually. The chart ships a bundled Postgres for demos; in production you turn it off and point at a managed instance:

postgresql:
  enabled: false       # do not run the bundled DB in prod

data:
  metadataConnection:
    host: bookshop-airflow.abc123.us-east-1.rds.amazonaws.com
    user: airflow
    db: airflow
    # password comes from a referenced Secret, not inline

pgbouncer:
  enabled: true
  maxClientConn: 100
  metadataPoolSize: 10
  resultBackendPoolSize: 5

Use managed RDS (or Cloud SQL on GCP) so backups, failover, and point-in-time restore are someone else’s cron job. Then turn on PgBouncer. Every scheduler, worker, triggerer, and api-server pod opens connections to Postgres, and with multiple schedulers and KEDA-scaled workers that count explodes past what Postgres wants to hold open. PgBouncer is a connection pooler that sits in front and multiplexes — the pods connect to it, it keeps a small warm pool to Postgres. On any deployment past a couple of pods it’s not optional; you’ll hit FATAL: sorry, too many clients already without it.

Scheduler HA and DAG delivery

Airflow 3.x supports multiple active schedulers running at once — real high availability, not active/passive. They coordinate through row-level locks (SELECT ... FOR UPDATE SKIP LOCKED) in the metadata database, so two schedulers never grab the same task, and if one pod dies the others keep scheduling without a failover pause:

scheduler:
  replicas: 2          # active/active HA in 3.x; both schedule concurrently

triggerer:
  replicas: 1          # runs deferred/async sensors (from the deferrable-operators post)

apiServer:
  replicas: 2          # the 3.x api-server + React UI; Flask webserver is gone

Two schedulers is the standard starting point — it survives a node loss and roughly doubles scheduling throughput. That’s also why the managed metadata DB matters: HA schedulers put more concurrent lock traffic on Postgres, which is exactly what PgBouncer absorbs.

The last big lever is how DAG code reaches the pods, and the chart supports the two mainstream answers. git-sync runs a sidecar that clones your DAG repo into a shared volume and re-pulls on an interval:

dags:
  gitSync:
    enabled: true
    repo: [email protected]:your-org/bookshop-airflow.git
    branch: main
    subPath: "dags"
    wait: 30           # seconds between pulls
    sshKeySecret: airflow-git-ssh

Git-sync gives you fast DAG-only iteration — push to main, and within wait seconds every pod has the new code without rebuilding an image. The tradeoff is that your dependencies (the requirements.txt) still come from the image, so a DAG that imports a new package needs an image rebuild anyway, and now you have two propagation paths to reason about. The other answer is the baked image: DAGs and dependencies together in defaultAirflowRepository:defaultAirflowTag, deployed by rolling the pods. One artifact, one propagation path, fully reproducible — at the cost of an image build and a pod roll for every DAG edit. I lean baked-image for the bookshop because reproducibility beats iteration speed once something pages a rotation, but git-sync earns its keep on teams shipping DAG tweaks hourly.

DAG deployment mechanics: how code actually propagates

Whichever platform you land on, “deploy a DAG” resolves to one of three delivery mechanisms, and their differences are entirely about propagation latency — how long between “merged” and “the scheduler sees it.”

  • Baked image. DAGs live inside the container image. Propagation = an image build plus a rolling pod restart, so minutes, and every pod is guaranteed identical. This is what astro deploy and a Helm image bump both do. Best reproducibility, slowest DAG-only loop.
  • git-sync. A sidecar re-clones on an interval (wait). Propagation = seconds to a minute, no rebuild. Fast, but code and dependencies now travel separately.
  • Object-storage sync (S3/GCS). Files land in a bucket; the platform pulls them into the DAGs folder. This is MWAA and Cloud Composer’s model. Propagation is the sync interval plus the scheduler’s own parse cadence.

That last term is the one people forget. The dag processor only rescans the DAGs folder every refresh_interval seconds (default 300 — five minutes):

# env: AIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL
[dag_processor]
refresh_interval = 30

Mind the name here if you’re coming from 2.x: this setting used to be [scheduler] dag_dir_list_interval, and it was renamed in Airflow 3.0 when DAG parsing moved out of the scheduler into the standalone dag processor. The old key still resolves as a deprecated alias, so a stale AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL in your .env will appear to work — which is exactly why the rename is worth knowing.

So on MWAA, a DAG synced to S3 can take S3 sync time + up to refresh_interval before it’s live. If your team keeps asking “I uploaded it, why isn’t it showing up,” this interval is usually the answer. Lowering it makes new DAGs appear faster at the cost of more parsing churn; don’t drop it to a few seconds on a deployment with hundreds of DAGs.

Native DAG versioning changes how this interacts with a rolling deploy. In 3.x each run records the DAG version it executed against, and the UI renders that run’s graph from the code as it was then. During a rolling deploy you briefly have old and new pods serving simultaneously — a task that started under version N and whose worker rolled mid-run won’t have its history rewritten to look like version N+1. The run stays pinned to the code it started with, which is exactly what you want when you’re staring at a failed backfill trying to figure out what actually executed. It doesn’t make deploys atomic — a DAG that changes shape mid-run can still surface edge cases — but it removes the old 2.x confusion where the graph always showed current code regardless of what ran.

What “config” means in production

Across all three managed roads, the same handful of things move from your laptop to the deployment. Your requirements.txt and the Docker image define the runtime — the Python packages and system libraries your DAGs import. Environment variables carry the settings that differ per environment. Connections and secrets move to a secrets backend (the one from the production-patterns post), so credentials live in Secrets Manager or its equivalent, not in the metadata database. And the whole thing is driven by CI/CD: the merge-to-main workflow teased at the end of the last post is where astro deploy — or an S3 sync, or a Composer bucket push — actually runs, so shipping a DAG is opening a PR and merging it, not SSHing anywhere. The local project and the production deployment are the same artifact; CI is the only thing that should touch prod.

MWAA in practice

MWAA hands you the same three delivery levers but wraps them in AWS’s opinions. You create an environment, point it at an S3 bucket, and pick an environment class that sizes the scheduler and workers together:

  • mw1.micro / mw1.small — dev and light workloads.
  • mw1.medium / mw1.large — production tiers with more scheduler CPU and headroom.

The bucket layout is fixed by convention: dags/ for your code, a requirements.txt for PyPI packages, and a plugins.zip for anything that has to land on the Python path as a plugin (custom operators, hooks, the airflow.plugins_manager extensions from the plugins post). MWAA also supports a startup_script.sh — a shell script it runs on every scheduler and worker before Airflow boots, which is where you set environment variables, export secrets, or install an OS package that isn’t a pip install:

#!/bin/bash
# startup_script.sh — runs on each MWAA component at startup
export DBT_PROFILES_DIR="/usr/local/airflow/dags/dbt"
export AIRFLOW__CORE__DAG_CONCURRENCY=32

You point the environment at these files by S3 object version — MWAA pins a specific version of requirements.txt, plugins.zip, and the startup script, so updating dependencies means uploading a new version and updating the environment, which triggers a rebuild that can take 20–30 minutes. That’s the heavy part of the MWAA loop. Autoscaling covers workers only: you set MinWorkers and MaxWorkers and MWAA scales the fleet on queued-task pressure, similar to KEDA but with less control. The real friction is version lag — MWAA supports specific Airflow versions on AWS’s timeline, so a 3.3.0 feature can be weeks or months from availability there. If you need the newest Airflow the day it ships, MWAA is not that road.

Cloud Composer in practice

Composer is Google’s managed Airflow on GKE, and the first thing to know is Composer 2 versus Composer 3. Composer 2 runs on a GKE cluster with autoscaling (via GKE node pools) that you have some visibility into. Composer 3 moves to a more fully managed model on GKE Autopilot, where Google runs the node infrastructure and you think in terms of workloads and resource requests rather than nodes — less to tune, less to see, and generally the one to pick for a new environment unless you have a specific reason to manage the cluster yourself.

DAG delivery is object-storage sync: Composer provisions a Cloud Storage bucket and syncs gs://<bucket>/dags/ into every component. Deploying is gsutil rsync or gcloud composer environments storage dags import into that bucket, then the same dag_dir_list_interval wait applies. Dependencies are PyPI packages you declare on the environment rather than a raw requirements.txt file:

gcloud composer environments update bookshop-prod \
  --location us-central1 \
  --update-pypi-packages-from-file requirements.txt

Composer resolves and installs those into the image, which — like MWAA — triggers an environment update that takes real time. Secrets go through Secret Manager, logs through Cloud Logging, and the Google operators (BigQuery, GCS, Dataproc) are first-class. Same bargain as MWAA on a different cloud: native integration and a managed control plane, paid for with a bucket-based deploy and a platform you tune less directly.

Astronomer in practice

Astronomer maps cleanly onto the tooling this whole series has used. The mental model is Workspaces (a team boundary — projects, users, RBAC) containing Deployments (a running Airflow environment: version, resources, environment variables, secrets backend). You provision a Deployment in the control plane, and shipping to it is the command you already know:

astro deploy <deployment-id>

That builds your Astro project’s image, pushes it, and rolls it out. The sharpest lever Astronomer adds is worker queues — named pools of workers with their own size and autoscaling, so you can route heavy tasks (a big dbt run) to a large-machine queue and keep light tasks on a cheap one, set per-task with the queue argument you’d use on any Celery setup. For fast iteration, Astronomer supports DAG-only deploys that skip the image rebuild:

astro deploy --dags       # ships only the dags/ folder, no image build — seconds, not minutes

That’s the git-sync tradeoff repackaged as a flag: use astro deploy --dags when you’ve only touched DAG code, and a full astro deploy when requirements.txt or the Dockerfile changed. For CI, you authenticate with a Deployment API token (or a Workspace token scoped to several Deployments) set as ASTRO_API_TOKEN in the environment, so the pipeline deploys without a human logging in. The trade for all this ergonomics is that it’s a paid product on top of open-source Airflow — you’re buying the platform team, not renting a cloud’s Airflow.

A real CI/CD deploy workflow

Whichever road you’re on, the shape is the same: a GitHub Actions workflow that runs on merge to main, and the only thing that differs is the final deploy step. Here’s the Astronomer version — test, then deploy on green:

# .github/workflows/deploy.yml
name: Deploy to Astronomer
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Astro CLI
        run: curl -sSL https://install.astronomer.io | sudo bash -s

      - name: Test DAGs parse
        run: |
          pip install -r requirements.txt
          python -m pytest tests/dags   # the DAG-integrity tests from the testing post

      - name: Deploy
        env:
          ASTRO_API_TOKEN: ${{ secrets.ASTRO_API_TOKEN }}
        run: astro deploy ${{ secrets.ASTRO_DEPLOYMENT_ID }}

Swap the last step and the same skeleton deploys to MWAA:

      - name: Sync DAGs to S3
        run: |
          aws s3 sync ./dags s3://bookshop-mwaa/dags --delete
          aws s3 cp ./requirements.txt s3://bookshop-mwaa/requirements.txt

…or to Cloud Composer:

      - name: Push DAGs to Composer bucket
        run: |
          gcloud composer environments storage dags import \
            --environment bookshop-prod --location us-central1 \
            --source ./dags

The invariant across all three: tests gate the deploy. The DAG-integrity test that imports every DAG and asserts no parse errors is the cheapest insurance you’ll ever buy — it catches the typo that would otherwise take down the whole scheduler’s parsing loop, and it runs in seconds before anything touches prod.

The 2.x → 3.x upgrade path

If you’re inheriting an existing 2.x deployment rather than starting fresh, the migration is real work but it’s a well-marked trail. Airflow ships a ruff-based ruleset that flags the deprecated constructs automatically. The AIR30x rules cover the renames and removals:

# find everything 3.x will reject
ruff check --select AIR30 dags/

# auto-fix the mechanical ones (import path rewrites, schedule_interval → schedule)
ruff check --select AIR30 --fix dags/

Most of what it finds is import paths and argument renames. The high-frequency ones:

2.x3.x
airflow.operators.pythonairflow.providers.standard.operators.python
airflow.operators.bashairflow.providers.standard.operators.bash
airflow.sensors.filesystemairflow.providers.standard.sensors.filesystem
from airflow import Datasetfrom airflow.sdk import Asset
schedule_interval= / timetable=schedule=
from airflow.decorators import dag, taskfrom airflow.sdk import dag, task
SubDAGsTaskGroups
sla=on_failure_callback / deadlines

Then the database itself migrates in place:

airflow db migrate      # applies the 3.x schema migrations; back up Postgres first

A migration checklist that won’t leave you stranded:

  1. Pin your versions. Freeze apache-airflow==2.x and every provider in requirements.txt so nothing floats mid-migration. You want one variable moving at a time.
  2. Run ruff --select AIR30 on the whole dags/ tree and fix everything — mechanical rewrites first with --fix, then the structural ones (SubDAGs → TaskGroups, sla= → callbacks) by hand.
  3. Snapshot the metadata database. RDS/Cloud SQL snapshot, or pg_dump. airflow db migrate rewrites schema; you want a restore point.
  4. Bump to 3.3.0 and the matching providers, run airflow db migrate against a copy first, and let it run in a staging environment.
  5. Re-test in staging — the DAG-integrity tests plus a real backfill of a representative day, so you exercise the scheduler-managed backfill path (which itself changed in 3.x).
  6. Roll production with the same image, and keep the old version’s image tag around so a rollback is one Helm/astro command, not a rebuild.

The removed features that most often force a code change are SubDAGs (gone entirely — the mechanical fix is TaskGroups, which you were probably already using) and SLAs (the sla= argument is gone; the replacement is on_failure_callback plus the newer deadline mechanism). Neither is a one-line rewrite, so budget for them rather than discovering them at migrate time.

The Airflow 3.x recap

You’ll spend the next few years reading Airflow material written for 2.x, and a lot of it is now actively wrong. Keep this list handy — these are the changes that will bite when you copy an old snippet:

  • Scheduling is schedule= only. schedule_interval= and timetable= are gone. One argument takes a cron string, a preset like @daily, a timedelta, or an Asset.
  • Datasets are now Assets. Same data-aware scheduling idea, renamed. from airflow.sdk import Asset.
  • Backfills are scheduler-managed. You trigger them from the UI or airflow backfill create and the scheduler runs them — no more launching a separate CLI process that owns the backfill on its own.
  • The Task SDK is the authoring home. from airflow.sdk import dag, task. DAGs talk to Airflow through the Task Execution API rather than reaching into the metadata database directly, which is what let Airflow split into a standalone api-server and a new React UI.
  • DAG versioning is native. The UI tracks which version of a DAG each run executed, so a run’s graph reflects the code as it was then, not as it is now.
  • SubDAGs and SLAs were removed. Use TaskGroups instead of SubDAGs; use on_failure_callback and the deadline mechanism instead of sla=.

If a tutorial uses schedule_interval, imports a SubDAG, or sets an sla, it predates 3.x. Trust the version, not the search ranking.

Final thoughts

Airflow’s real job was never running your Python — cron can run Python. Its job is to know what ran, when, against which slice of data, and what to do when a piece of it fails, and to keep knowing all of that whether the run is a scheduled daily or a backfill of last year. That’s the difference between a pile of scripts and an orchestrator. Once it’s deployed, Airflow orchestrates the movement — but the transformations themselves are their own craft, and the dbt series is where that lives: the SQL models Airflow schedules, treated as real software. If you want to see how the other orchestrators frame the same problem, Dagster’s asset-first model and Prefect’s lighter, Python-native take are both worth a weekend — knowing where Airflow’s choices are load-bearing and where they’re just history is what makes you good at this. That’s the deployment story end to end. What’s left in this series is the machinery you reach for once the pipeline is live and load-bearing: conditional flow and typed params, custom XCom backends, secrets in depth, the REST API, plugins and listeners, scheduler tuning, and Kubernetes.

Next: Branching, Setup/Teardown, and Params

Comments