Your Airflow, Running in One Command

Airflow is not one process. It’s a scheduler, a dag-processor, an api-server, a triggerer, a metadata database, and workers — a small distributed system you now have to run on your laptop. The naive way is to fight a docker-compose.yaml with a dozen services and a page of environment variables, or to run airflow standalone and get a single-process toy that behaves nothing like production. There’s a better default for learning, and it’s one command. But “one command” is a choice, and it’s worth understanding what you’re choosing over before you type it.

The honest alternatives matrix

There are four realistic ways to run Airflow 3.x on your own machine. None is wrong; they trade different things.

A plain uv (or pip) virtualenv. You install Airflow into a Python environment and run the pieces yourself:

uv venv && source .venv/bin/activate
uv pip install "apache-airflow==3.3.0" \
  --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.3.0/constraints-3.12.txt"
airflow db migrate
airflow standalone

That constraint URL matters: Airflow has a large dependency tree, and without pinning transitive versions the install resolves to something that may not run. This is the lightest option — no Docker, fast startup, and you can drop into a debugger against the real process. The cost is that you’re now the one keeping the metadata database, the scheduler, and every provider’s system dependency straight, and you’re running on SQLite with the sequential executor unless you stand up Postgres yourself. Good for reading Airflow’s own source or hacking on a provider; a poor model of production.

airflow standalone. The single command above runs the scheduler, dag-processor, api-server, and triggerer in one process against a SQLite database. It’s the fastest way to see a UI. It’s also the worst way to learn the shape of Airflow: SQLite forbids parallel writes, so you’re locked to the sequential executor, one task at a time, and nothing about that resembles what you’ll deploy. Fine for a thirty-second sanity check; misleading as a daily driver.

The official docker compose. Airflow publishes a docker-compose.yaml that brings up the real service split — separate scheduler, dag-processor, api-server, triggerer, worker, Postgres, and Redis — using the CeleryExecutor. This is production-shaped and completely vendor-neutral. The catch is that the compose file is yours to babysit: a hundred-plus lines of service definitions, health checks, AIRFLOW__* variables, and a mandatory AIRFLOW_UID you have to set or your logs come out owned by root. Every version bump means reconciling your edits against a new upstream file. It’s the right tool if you refuse to depend on a vendor’s wrapper, and a lot of manual upkeep otherwise.

The Astro CLI. Astronomer — the company behind much of Airflow’s development — ships a free command-line tool that scaffolds a project and runs that same real service split in Docker, but hides the compose file behind sensible commands. You get production shape without hand-editing YAML, plus a project layout that deploys unchanged to a real host later.

This series uses the Astro CLI, and the reason is narrow: it’s the only option that gives a newcomer the correct mental model — separate services, Postgres, LocalExecutor running tasks in parallel — with none of the yak-shaving, while keeping the local project byte-for-byte the thing you ship. standalone lies about the architecture; the bare venv makes you the platform team; raw compose makes you a YAML mechanic. The CLI is the one that lets you spend your attention on DAGs. If you have a hard no-vendor-tooling rule, reach for the official compose file — everything you learn here transfers, because underneath, the Astro CLI is a compose file.

Installing the CLI

Install it with Homebrew on macOS or the install script on Linux:

brew install astro

You’ll need Docker running. That’s the only real prerequisite: the CLI pulls a runtime image that already contains a matching Airflow, so you don’t install Airflow itself by hand. Docker Desktop is the common choice, but the CLI is happy against any Docker-compatible runtime — Colima or Podman on macOS work fine if you’d rather not run Desktop — as long as the CLI can find a Docker socket. Confirm the install:

astro version

Scaffold a project

Make a directory for the bookshop pipeline and initialize it:

mkdir bookshop-airflow && cd bookshop-airflow
astro dev init

That lays down a working project:

bookshop-airflow/
├── dags/                  # your DAG files live here
│   └── example_astronauts.py
├── include/               # SQL, scripts, anything your DAGs import
├── tests/                 # DAG tests
├── Dockerfile             # pins the Airflow runtime image
├── requirements.txt       # Python deps layered onto the image
├── packages.txt           # OS-level packages
├── airflow_settings.yaml  # local connections/variables/pools (git-ignored)
└── .env

The two files you’ll touch most are dags/, where every DAG you write goes, and requirements.txt, where you add Python libraries — DuckDB, requests, a provider package — that get baked in on the next start. include/ is for the non-DAG assets your tasks need — a SQL file, a lookup CSV — kept out of dags/ so the dag-processor doesn’t try to parse them as Python.

The runtime image and how versions map

The Dockerfile’s first line is the one that decides which Airflow you’re running:

FROM astrocrpublic.azurecr.io/runtime:3.1-1

That’s not an Airflow version — it’s an Astro Runtime tag, a Debian image with a specific Airflow build, the standard providers, and Astronomer’s patches already inside. Runtime tags and Airflow versions move on separate tracks, so you have to read the mapping. A tag like 3.1-1 means Astro Runtime 3.1, patch 1, which ships a particular Airflow 3.3.x. The rule of thumb: the major of the Runtime tracks a line of Airflow releases, and the number after the dash is Astronomer’s patch on top. When you need the exact Airflow inside an image, don’t guess — ask it:

astro dev bash -s scheduler
airflow version      # prints the Airflow version baked into this image

It helps to picture the image as layers. The Runtime tag is the base — Airflow, the standard providers, Python, Debian. On top of that, the build installs whatever’s in packages.txt with apt (system libraries: a database client, git, a compression tool) and then whatever’s in requirements.txt with pip (Python packages: DuckDB, a provider, requests). Your dags/ and include/ folders aren’t baked in at all locally — they’re bind-mounted, which is why editing a DAG doesn’t need a rebuild but editing requirements.txt does. That layering is also why builds are usually fast: Docker caches the base and the package layers, so a one-line change to requirements.txt only rebuilds from that layer down, not from scratch.

Pinning and upgrading. You pin by writing an explicit tag, which astro dev init already does — never leave it floating on latest, or a teammate’s astro dev start next month builds a different Airflow than yours. To upgrade, change the tag and rebuild:

# was: FROM astrocrpublic.azurecr.io/runtime:3.1-1
FROM astrocrpublic.azurecr.io/runtime:3.2-1

astro dev restart      # rebuilds against the new image

astro dev upgrade-test will dry-run a version bump and flag dependency conflicts before you commit to it — worth running against anything with a real DAG in it, because a Runtime bump can pull new provider versions that change import paths. Pin deliberately, upgrade on purpose, and the “works on my machine, breaks in CI” class of bug mostly disappears.

Bring it up

astro dev start

The first run builds the image, so give it a minute. When it settles you have the full 3.x stack up: the scheduler, the dag-processor (a separate process in 3.x — it parses your DAG files so the scheduler doesn’t have to, which is exactly why a parse error shows up there and not in the scheduler), the api-server, the triggerer, and a Postgres metadata database. Same set of moving parts from the last post, now running locally, with the LocalExecutor so tasks actually run in parallel.

Open localhost:8080 and log in with admin / admin. The 3.x React UI loads with a couple of bundled example DAGs already parsed — real, runnable graphs you can trigger to confirm the whole stack is wired up before you’ve written a line. Toggle one on, watch it run, then leave the examples be; you’ll replace them with the bookshop DAG next.

The everyday loop

The development loop is tight because dags/ is mounted into the containers. Save a file and the dag-processor re-parses it within seconds — new DAGs show up in the UI on their own, no restart. You spend your day editing files and refreshing the browser.

The handful of commands worth memorizing:

astro dev restart   # rebuild + restart — after editing requirements.txt or Dockerfile
astro dev logs      # tail logs across the containers (-f to follow)
astro dev stop      # stop everything, keep the metadata DB
astro dev kill      # tear down and wipe state for a clean slate

The rule of thumb: DAG code hot-reloads on its own, but anything that changes the image — a new line in requirements.txt, a Dockerfile edit — needs a restart to take effect. If a freshly-imported library throws ModuleNotFoundError, you added it to requirements.txt but haven’t restarted yet.

You don’t always need the browser, either. Once the stack is up you can drive Airflow from inside the container, which is faster for a quick check than clicking through the UI:

astro dev bash                                   # shell into the scheduler
airflow dags list                                # every parsed DAG
airflow dags test bookshop_daily 2026-05-15      # run one full DAG run, now, in the foreground
airflow tasks test bookshop_daily load 2026-05-15  # run a single task, logs to your terminal

airflow tasks test is the tightest feedback loop there is — it runs one task in-process, prints its logs straight to the terminal, and writes nothing to the metadata database, so you can hammer on a single step without scheduling a whole run. It’s the local equivalent of a print-and-run cycle, and it’s where you’ll live while debugging a task in the next few posts.

Configuration: .env and airflow_settings.yaml

Two files carry your local configuration, and they do different jobs.

.env sets Airflow config and secrets via environment variables. Every setting in Airflow’s config has an environment-variable form: AIRFLOW__<SECTION>__<KEY>, uppercase, with the double underscores separating section from key. Want the UI in your timezone, or a faster pickup of new DAG files while you’re iterating? Set it in .env:

# .env
AIRFLOW__CORE__DEFAULT_TIMEZONE=America/New_York
AIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL=30      # rescan dags/ every 30s (default: 300)
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION=True
AIRFLOW__SCHEDULER__CREATE_CRON_DATA_INTERVALS=True

That last line is the important one, and it will make no sense until the scheduling chapter — so take it on faith for now and read the explanation there. The one-sentence version: Airflow 3 changed what a scheduled run means, and this restores the windowed-batch model that every DAG in this book assumes. Leave it out and your date filters will quietly match zero rows.

Each line maps to a row in Airflow’s config file — [core] default_timezone, [dag_processor] refresh_interval — but the environment-variable form wins over the file and, crucially, is the same mechanism a production host uses. Set it here, set it identically in the deployment, and the two environments behave the same. .env is git-ignored by default, which is what makes it the right home for anything secret.

airflow_settings.yaml declares connections, variables, and pools so you don’t click them into the UI by hand every time you wipe the database. This is a local-only convenience file (git-ignored), and it’s how the bookshop’s DuckDB connection gets recreated automatically on every astro dev start:

airflow:
  connections:
    - conn_id: bookshop_duckdb
      conn_type: duckdb
      conn_host: /usr/local/airflow/include/bookshop.duckdb
  variables:
    - variable_name: orders_source_url
      variable_value: https://example.com/bookshop/orders.csv
  pools:
    - pool_name: warehouse
      pool_slot: 4
      pool_description: cap concurrent warehouse hits

Connections are credentials-plus-endpoint for the things your tasks talk to (post 8 goes deep on these); variables are runtime key/value config your DAGs read; a pool caps how many tasks can hit a shared resource at once — here, at most four tasks touch the warehouse simultaneously no matter how many are ready. On startup the CLI reads this file and materializes all three into the metadata database. Because it holds real credentials, it never gets committed — the git-ignore is deliberate, not an oversight.

When you’re unsure what a setting actually resolved to — because config can come from the image’s defaults, the config file, or an .env override, in that order of increasing precedence — ask Airflow rather than reasoning it out:

astro dev bash
airflow config get-value core default_timezone   # what's effective right now

That prints the winning value after all the layers are applied, which settles most “is my .env even being read” questions in one line.

The wider CLI

start / stop / restart are the daily three, but the CLI does more, and a few commands save real time:

astro dev bash                    # shell into the scheduler container
astro dev bash -s dag-processor   # ...or a specific container
astro dev pytest                  # run tests/ inside the Airflow environment
astro dev parse                   # parse every DAG, fail on import errors
astro dev object export           # dump current connections/variables/pools to airflow_settings.yaml
astro dev object import           # push airflow_settings.yaml into the running DB

astro dev parse is the one to wire into muscle memory: it loads every file in dags/ in the real image and exits non-zero on the first import error, which turns “why isn’t my DAG showing up” into an immediate stack trace. astro dev pytest runs your tests/ folder inside the container, against the same Airflow the DAGs run under, so a test that passes locally isn’t passing against some other version on your host. And astro dev object export is the mirror of the settings file — if you did click a connection into the UI, this writes it back out to airflow_settings.yaml so it survives the next kill.

A few flags worth knowing. astro dev start -p 8081 moves the UI off the default port when 8080 is taken. astro dev start --wait 3m extends how long the CLI waits for the stack to report healthy before it gives up — useful on a cold machine where the first image build is slow. And when a rebuild insists on serving a stale layer — you bumped a pin but pip keeps installing the old version because Docker cached the layer — astro dev restart --no-cache forces a clean build from the base image down. It’s slow, so reach for it only when the cache is clearly lying to you, but it’s the fix when a dependency change stubbornly refuses to take. And when the metadata database gets into a state you don’t want to reason about — a half-migrated DAG, a stuck run, an experiment you’d rather forget — astro dev kill tears everything down including the Postgres volume, so the next start comes up on a clean database. It’s the “have you tried turning it off and on again” of local Airflow, and it works.

When it doesn’t come up

A few failures are common enough to name:

Port 8080 is already in use. Something else — often a stray container or a previous astro project you forgot to stop — holds the port. Either free it (astro dev stop in the other project) or move this one: astro dev start -p 8081.

Docker runs out of memory. The full stack wants headroom; on Docker Desktop’s stingy default, containers get OOM-killed and the UI never settles. Give Docker at least 4 GB (Settings → Resources). The tell is a scheduler that keeps restarting in astro dev logs with no Python error to explain it — that’s the kernel killing it, not your code.

The image won’t build. A bad line in requirements.txt — a package that doesn’t exist, or two pins that conflict — fails the build, and the error is in the astro dev start output, not hidden. Read the pip resolver’s complaint, fix the pin, and astro dev restart. A system library that needs an OS package goes in packages.txt, not requirements.txt.

A DAG doesn’t appear in the UI. Almost always a parse error. The dag-processor tried to import the file, hit an exception, and skipped it — the UI shows an import-errors banner, and astro dev logs (or the dag-processor logs specifically) has the traceback. Run astro dev parse to reproduce it on demand. If there’s no error and the file still won’t show, check that it’s actually in dags/ (not include/), that it defines a DAG at module top level, and give the dag-processor its refresh interval — a few seconds — to notice a brand-new file.

The metadata DB drifted. Switch branches, rename a DAG, delete one — and the UI can hold onto ghosts: a paused DAG that no longer exists in code, a run stuck in queued, a connection you thought you removed. The metadata database remembers state your files no longer describe. Small drift is normal and harmless; when it stops making sense, astro dev kill followed by astro dev start gives you a fresh database, and airflow_settings.yaml rebuilds your connections and pools automatically so the reset costs you nothing you’ll miss.

The through-line for all of these: astro dev logs -f is the first place to look, not the last. Airflow’s failures are almost always written down somewhere — a traceback in the dag-processor, an OOM in the scheduler, a pip error in the build — and the logs are where they’re written.

What you build locally is what you ship

This is the payoff, and it’s worth making concrete. The project astro dev init scaffolded — the Dockerfile with its pinned Runtime tag, requirements.txt, packages.txt, dags/ — is not a local-dev approximation of production. It is the deployment artifact. When it’s time to run this somewhere that pages a rotation instead of you, the same directory ships:

astro deploy

That builds the exact image you’ve been running, pushes it, and rolls it out to a managed Airflow. The .env overrides become the deployment’s environment variables; the connections you declared move to a secrets backend; nothing about the DAGs changes, because the runtime is identical down to the Airflow patch version. The alternatives don’t give you this — a standalone SQLite setup or a hand-tuned compose file is a thing you’d have to re-create for production. Here, the local project and the shipped project are one artifact, which is the whole reason the day this stops being a toy isn’t a rewrite.

In a real team you rarely type astro deploy by hand — it runs in CI, triggered by a merge to main. Because the whole project is just files in a git repo (Dockerfile, requirements.txt, dags/), shipping a DAG becomes opening a pull request: a reviewer reads the diff, CI runs astro dev pytest and astro dev parse to catch a broken import before it reaches production, and the merge is what deploys. That’s the same pytest/parse you’ve been running locally, now standing guard on the way to prod — which only works because local and CI and production are the same image. We’ll walk the full deploy path — Astronomer, MWAA, Cloud Composer, and what each one costs — in the advanced series’ Shipping It.

Final thoughts

Getting Airflow running used to be the tax you paid before learning any of it — an afternoon lost to compose files and version mismatches, the kind of yak-shave that makes people give up before the interesting part. Collapsing that to astro dev init and astro dev start isn’t a convenience so much as a decision to spend your attention on DAGs instead of on Docker. The stack underneath is still five services and a database; you just don’t have to hand-wire them to start thinking in graphs, and — this is the part that matters later — the thing you wired up locally is the thing that deploys. Next post, you write one.

Next: Your First DAG: Tasks That Know Their Order