Connections, Variables, and Secrets Backends

How Airflow stores external configuration, when to use Variables, how hooks read Connections, and how secrets backends change the lookup path.

Back in the connections-and-hooks chapter we drew a line: Connections are for keys, Variables are for knobs. That line is the right mental model, and it’s enough to write a working DAG. But it leaves three questions unanswered, and every one of them turns into a production incident if you get it wrong. Where does the credential actually live when the worker asks for it? What order does Airflow search when the same id exists in two places? And how do you keep the whole thing out of git without making local development miserable?

This chapter is the answer to all three. It’s the last stop in the foundations series, and it’s the one that separates “my DAG runs on my laptop” from “my DAG runs in production without a password ever touching the repo.”

Connections, three ways — and the trade-offs

A connection is a named bundle: connection type, host, login, password, schema, port, and an open-ended extra field for everything else the system wants. You already know it has an id and that your DAG names the id, not the secret. What’s worth slowing down on is that there are three places that bundle can come from, and they’re not interchangeable — each is right for a different phase of a project.

The UI (and airflow_settings.yaml). In the Airflow UI, Admin → Connections → add gives you a form: pick a type, fill in the fields, save. The connection lands as an encrypted row in the metadata database. This is fine for a human-managed connection you’ll rarely change, but a form isn’t reproducible — if you rebuild the environment, someone has to re-enter it by hand.

The Astro CLI fixes that for local dev with airflow_settings.yaml at your project root. It’s a checked-in file that declares connections, variables, and pools, and the local stack loads them on startup so you never re-type them:

airflow:
  connections:
    - conn_id: bookshop_db
      conn_type: postgres
      conn_host: db.internal
      conn_login: bookshop
      conn_password: s3cr3t
      conn_schema: orders
      conn_port: 5432
      conn_extra: '{"sslmode": "require"}'
  variables:
    - variable_name: bookshop_batch_size
      variable_value: "500"

Reproducible, yes — but note conn_password sits in plaintext. That’s acceptable for a throwaway local database and a fake secret. It is not acceptable for anything real, which is exactly why airflow_settings.yaml belongs in .gitignore the moment a value stops being a toy. Keep a airflow_settings.yaml.example with placeholders in the repo instead.

The environment variable, in URI form. Airflow reads any AIRFLOW_CONN_<ID> variable and treats it as a connection — the id is the suffix, lowercased. The value is a connection URI that packs everything into one string:

export AIRFLOW_CONN_BOOKSHOP_DB='postgresql://bookshop:[email protected]:5432/orders?sslmode=require'

That defines bookshop_db with no database row and no UI click. The ?sslmode=require query string becomes the connection’s extra. This is the form you’ll use inside containers and CI, where a .env line or an injected environment variable is the natural home for config.

The URI form has one genuine pain: if your password contains @, /, or :, you have to percent-encode it, and a mis-encoded URI fails in confusing ways. Airflow 3 lets you sidestep that with a JSON form of the same variable, which is far easier to read and to template:

export AIRFLOW_CONN_BOOKSHOP_DB='{
  "conn_type": "postgres",
  "host": "db.internal",
  "login": "bookshop",
  "password": "p@ss/w:rd",
  "schema": "orders",
  "port": 5432,
  "extra": {"sslmode": "require"}
}'

Same connection, no encoding gymnastics, and extra is a real nested object instead of a query string.

A secrets backend, for production. You point Airflow at Vault, AWS Secrets Manager, or GCP Secret Manager, and it resolves bookshop_db there. Rotation, audit, and access control become a solved problem owned by your platform team instead of rows you manage. The DAG still just says bookshop_db — only the storage moves. This is the big one, and it gets its own section below.

extra, extra_dejson, and where the provider settings go

The extra field is the part newcomers underuse. It’s a JSON blob attached to a connection for provider-specific settings that don’t fit host/login/password — SSL mode, a warehouse and role for Snowflake, an AWS region, an API’s auth scheme, connection timeouts. Hooks read it back as a parsed dict via extra_dejson:

from airflow.sdk import BaseHook

conn = BaseHook.get_connection("bookshop_db")
conn.host            # "db.internal"
conn.extra_dejson    # {"sslmode": "require"}
sslmode = conn.extra_dejson.get("sslmode", "prefer")

The rule that follows: settings that describe how to reach a system belong in extra, not in DAG code. If you find yourself writing sslmode="require" as a literal in a task, that’s a value that should have travelled with the connection. Put it in extra and every task that opens bookshop_db inherits it for free.

Fernet, and what it does and doesn’t protect

When a connection or variable lands in the metadata database, Airflow encrypts the sensitive fields at rest using a Fernet key — a symmetric key set via AIRFLOW__CORE__FERNET_KEY. Generate one once and treat it as infrastructure:

from cryptography.fernet import Fernet
print(Fernet.generate_key().decode())
export AIRFLOW__CORE__FERNET_KEY='fUj...your-generated-key...=='

Two things people learn the hard way. First, if you lose the key, every encrypted secret in the metastore becomes unrecoverable ciphertext — back it up like you’d back up a database. Second, Fernet only protects secrets at rest in Airflow’s own database. It does nothing about who can read that database, nothing about rotation, and nothing about secrets that live in airflow_settings.yaml or a .env in plaintext. Fernet is a floor, not a strategy — which is the whole argument for secrets backends.

Variables, done right

A Variable is a small piece of non-secret runtime config: a batch size, a target table name, a feature flag, a list of regions. In the Task SDK you read one with Variable.get, and you can pull JSON straight into a Python object:

from airflow.sdk import Variable

batch = Variable.get("bookshop_batch_size", default=500)
regions = Variable.get("bookshop_regions", deserialize_json=True)   # ["us", "eu"]

Setting one from code is symmetrical, and serialize_json=True writes a structure back as JSON:

Variable.set("bookshop_last_full_load", "2026-07-09")
Variable.set("bookshop_regions", ["us", "eu", "apac"], serialize_json=True)

In templates — the Jinja fields from the previous chapter — Variables are exposed two ways. {{ var.value.<key> }} gives the raw string; {{ var.json.<key> }} parses a JSON variable and lets you index into it:

BashOperator(
    task_id="report",
    bash_command="generate_report --batch {{ var.value.bookshop_batch_size }} "
                 "--region {{ var.json.bookshop_regions[0] }}",
)

And like connections, Variables have an environment-variable form: AIRFLOW_VAR_<KEY>, uppercased. export AIRFLOW_VAR_BOOKSHOP_BATCH_SIZE=500 defines bookshop_batch_size with no database row — the same container-friendly pattern as AIRFLOW_CONN_*. Note that AIRFLOW_VAR_* values are always strings, so a JSON variable set this way still needs deserialize_json=True on the read.

One behavior that bites newcomers: Variable.get("missing_key") with no default raises — in Airflow 3 it’s an AirflowRuntimeError carrying a VARIABLE_NOT_FOUND error type — and fails the task. That’s usually what you want: a missing required config should be loud, not silent. But it means default isn’t optional politeness; it’s how you distinguish “this variable is genuinely optional” from “this variable must exist.” Pass default=500 when a sensible fallback exists, and omit it when a missing value should stop the run. Don’t reach for a bare try/except around the call to paper over a config you forgot to set.

(Watch the keyword. In Airflow 2 it was default_var; the Task SDK’s signature is Variable.get(key, default=..., deserialize_json=False). Most tutorials you’ll find still spell it the old way.)

The top-level-parse anti-pattern

Here’s the mistake that quietly degrades a whole deployment. The scheduler re-parses every DAG file constantly — that’s how it discovers changes. Any code at the top level of a DAG file (module scope, outside a task) runs on every parse. So this:

# DON'T: runs on every single DAG parse
from airflow.sdk import Variable
BATCH = Variable.get("bookshop_batch_size")   # a database hit, every parse, forever

with DAG("bookshop"):
    ...

turns your config lookup into a load-bearing part of scheduler performance. Multiply one Variable.get by hundreds of parses per minute across dozens of DAGs and you’ve built a self-inflicted denial-of-service against your own metadata database — or, with a secrets backend, against AWS Secrets Manager, where you’re also now paying per API call.

The fix is to read Variables where they’re actually needed: inside a task, which only runs when the task runs, or in a template, which only renders at execution time. Both defer the lookup out of the parse loop:

@task
def load_orders():
    batch = Variable.get("bookshop_batch_size", default=500)   # runs at task time, not parse time
    ...

If you truly need a value at parse time to shape the DAG’s structure, that value is a deployment concern, not a runtime one — put it in an environment variable and read it with os.getenv, which is a cheap in-process lookup with no database or API behind it.

Secrets backends, in full

A secrets backend is a pluggable source that Airflow consults for connections and variables before falling back to its own database. You configure it with two settings — the backend class and its keyword arguments:

export AIRFLOW__SECRETS__BACKEND='airflow.secrets.local_filesystem.LocalFilesystemBackend'
export AIRFLOW__SECRETS__BACKEND_KWARGS='{"connections_file_path": "/opt/airflow/secrets/connections.yaml", "variables_file_path": "/opt/airflow/secrets/variables.yaml"}'

backend_kwargs is JSON, and its keys are specific to the backend class. The two you’ll set on almost every real backend are connections_prefix and variables_prefix — the path or name prefix under which the backend looks for each. This lets one Vault or Secrets Manager instance hold everything, with Airflow’s secrets namespaced under a known root.

The resolution order

This is the single most important fact in the chapter, because it explains every “why is it using the wrong password” mystery. When a hook asks for connection bookshop_db, Airflow searches in a fixed order and returns the first hit:

  1. Environment variables — AIRFLOW_CONN_BOOKSHOP_DB (or AIRFLOW_VAR_* for variables)
  2. The configured secrets backend — Vault, Secrets Manager, etc.
  3. The metadata database — the UI/airflow_settings.yaml rows

Environment always wins. That’s a feature: it means you can override any connection locally by exporting AIRFLOW_CONN_*, without touching the backend or the database. It’s also a trap: if a stale AIRFLOW_CONN_BOOKSHOP_DB is lingering in a worker’s environment, it silently shadows the correct value in Secrets Manager, and nothing in the UI will show you why. When a connection resolves to something you didn’t expect, check the environment first, the backend second, the database last — in that order, because that’s the order Airflow used.

LocalFilesystemBackend for dev

LocalFilesystemBackend reads connections and variables from local YAML or JSON files. It’s the honest way to develop against a secrets backend without standing up Vault: your DAG code and resolution path are identical to production, only the storage is a file. A connections.yaml:

bookshop_db:
  conn_type: postgres
  host: db.internal
  login: bookshop
  password: s3cr3t
  schema: orders
  port: 5432
  extra:
    sslmode: require

and a variables.yaml:

bookshop_batch_size: "500"
bookshop_regions: '["us", "eu"]'

These files hold secrets in plaintext, so the same rule as airflow_settings.yaml applies: they never go in git.

A worked AWS Secrets Manager setup

For production on AWS, the backend is SecretsManagerBackend from the Amazon provider. Configure it once:

export AIRFLOW__SECRETS__BACKEND='airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend'
export AIRFLOW__SECRETS__BACKEND_KWARGS='{
  "connections_prefix": "airflow/connections",
  "variables_prefix": "airflow/variables",
  "region_name": "us-east-1"
}'

With those prefixes, a connection named bookshop_db is looked up at the secret airflow/connections/bookshop_db, and a variable bookshop_batch_size at airflow/variables/bookshop_batch_size. You create the connection secret with the same URI (or JSON) form you’d use in an environment variable:

aws secretsmanager create-secret \
  --name airflow/connections/bookshop_db \
  --secret-string 'postgresql://bookshop:[email protected]:5432/orders?sslmode=require'

and a variable is just its plain value:

aws secretsmanager create-secret \
  --name airflow/variables/orders_api_key \
  --secret-string 'sk_live_9f2a...'

Two operational notes worth internalizing early. Set connections_prefix (or variables_prefix) to null to tell Airflow not to search Secrets Manager for that kind at all — useful when you keep connections in the backend but variables in the database, and want to avoid a failed lookup and its cost on every miss. And workers authenticate to Secrets Manager with the usual AWS credential chain (an IAM role on the task, typically), so the DAG carries no AWS keys either — the secret store’s own auth replaces them.

The rotation payoff is the reason platform teams push for this. When the bookshop database password changes, you rotate the value in Secrets Manager once — aws secretsmanager put-secret-value — and every worker picks up the new credential on its next lookup, because nothing cached the old one in a DAG file, a .env, or a metastore row. No redeploy, no restart, no editing twelve DAGs. Compare that to a password pasted into code: rotation there means a code change, a review, and a deploy, for something that should be an ops action. The indirection you set up in the connections chapter is exactly what makes rotation a non-event.

Airflow can also resolve its own config values from the same backend — a [core] sql_alchemy_conn or a [smtp] smtp_password can be stored as a secret with a config_prefix and pulled in the same lookup path. You won’t need that on day one, but it’s the same machinery, and it means the pattern in this chapter scales all the way down to Airflow’s own bootstrap secrets.

The same shape in Vault

HashiCorp Vault is the same pattern with a different provider. The class is VaultBackend, and the kwargs point at your mount and paths:

export AIRFLOW__SECRETS__BACKEND='airflow.providers.hashicorp.secrets.vault.VaultBackend'
export AIRFLOW__SECRETS__BACKEND_KWARGS='{
  "connections_path": "connections",
  "variables_path": "variables",
  "mount_point": "airflow",
  "url": "https://vault.internal:8200"
}'

A connection bookshop_db then lives at the KV path airflow/connections/bookshop_db, with the fields stored under a conn_uri key. The mechanics differ; the mental model doesn’t. This chapter is the intro-level tour — the Airflow in Practice series has a deeper secrets-backends chapter that gets into multiple backends, auth methods, and caching.

Keeping secrets out of the logs

Resolving a secret cleanly at run time is only half the job. The other half is making sure it doesn’t leak on the way through — into a task log, an XCom, or the Rendered Template view from the previous chapter. A pipeline that reads a password from Vault and then prints it in a log line has undone all of the work above.

Airflow helps here with secret masking. Any value that came from a connection’s password field, or from a Variable whose name contains a sensitive-looking token — password, secret, api_key, token, and a few others — is automatically redacted to *** wherever Airflow renders it: log output, the UI’s Variable list, the Rendered Template panel. That’s why naming matters. A Variable called orders_api_key gets masked; the identical value stashed in one called orders_config does not. Name secrets like secrets and Airflow will hide them for you.

Masking is a safety net, not a license to be careless. It only catches values Airflow knows are sensitive, and it only catches them in Airflow’s own rendering — it can’t reach inside a print() you wrote or a third-party library’s debug output. So two habits pay off: give sensitive Variables telltale names so the automatic masking engages, and never return a credential from a task (that puts it in XCom, where it’s stored and displayed). Read the secret, use it, let it fall out of scope. In the bookshop’s fetch_orders, api_key lives entirely inside the task and never becomes a return value — that’s deliberate.

One more edge from the connections form: in Airflow 3, the “Test” button on the connection editor is disabled by default, because testing a connection means the server dials out to whatever host you typed. If you want it in a trusted environment, you opt in with AIRFLOW__CORE__TEST_CONNECTION=Enabled — but the default-off posture is the right instinct for anything internet-facing.

The decision: Variable, Connection, or secret

With all three stores on the table, the choice is usually clear once you ask two questions.

Is it a credential or an endpoint — something that reaches an external system? Then it’s a Connection, resolved from a secrets backend in production. Database passwords, API base URLs and their auth, bucket clients: connections, always.

Is it non-secret runtime config — a knob a human might turn without a code change? Then it’s a Variable: batch sizes, feature flags, region lists, a target table name.

Is it a secret that isn’t a connection — a lone API token, a signing key, a webhook secret? It’s still config-shaped like a Variable, but because it’s sensitive, serve it through the secrets backend rather than the plaintext Variable store. A Variable resolved from Secrets Manager is a secret with a Variable’s ergonomics.

And the value that shapes the DAG’s structure at parse time — how many mapped tasks, which branch exists — isn’t any of the three. That’s a plain environment variable read with os.getenv, because it must be cheap and it must be available before a task ever runs.

Underneath all four is one rule the whole chapter serves: if you can grep a secret out of your repository, you’ve already lost. Code names a resource; it never embeds one.

Wiring the bookshop end to end

Here’s the daily bookshop load with both a Postgres connection and an API key, every secret resolved at run time from the backend and nothing sensitive in the file:

from airflow.sdk import DAG, task, Variable
from airflow.providers.http.hooks.http import HttpHook
from airflow.providers.postgres.hooks.postgres import PostgresHook

with DAG("bookshop_daily", schedule="@daily", catchup=False):

    @task
    def fetch_orders(data_interval_start):
        # orders_api connection -> Secrets Manager: airflow/connections/orders_api
        # orders_api_key variable -> Secrets Manager: airflow/variables/orders_api_key
        api_key = Variable.get("orders_api_key")            # resolved at task time, not parse time
        hook = HttpHook(method="GET", http_conn_id="orders_api")
        day = data_interval_start.strftime("%Y-%m-%d")
        resp = hook.run(
            endpoint=f"/orders/{day}.json",
            headers={"Authorization": f"Bearer {api_key}"},
        )
        return resp.json()

    @task
    def load_orders(rows, data_interval_start):
        # bookshop_db connection -> Secrets Manager: airflow/connections/bookshop_db
        hook = PostgresHook(postgres_conn_id="bookshop_db")   # extra_dejson sslmode travels with it
        day = data_interval_start.strftime("%Y-%m-%d")
        hook.run("DELETE FROM orders WHERE order_date = %s", parameters=(day,))
        hook.insert_rows(table="orders", rows=rows)

    load_orders(fetch_orders())

Read the file top to bottom and there is not one host, password, token, or region in it. orders_api and bookshop_db are ids; orders_api_key is a name. At parse time the scheduler touches none of them — no top-level Variable.get, no connection lookup outside a task. When the task actually runs, Variable.get("orders_api_key") and the two hooks each walk the resolution order, hit Secrets Manager, and return live values. sslmode=require rides along in the connection’s extra, so the task never has to know the database wants SSL.

Now change environments. Locally, you swap the backend for LocalFilesystemBackend (or export AIRFLOW_CONN_* / AIRFLOW_VAR_* in your .env, which win outright) and point orders_api at a mock. In production the backend is Secrets Manager and the same ids resolve to the real warehouse and the real API. The DAG file is byte-for-byte identical across both. That portability is the entire point — and it’s why this, the last piece of the foundations, is the piece that makes everything before it deployable.

Examples in this chapter are docs-checked against the series’ stated Airflow 3.3.0 version, but they were not executed in this repository.

Final thoughts

Connections, Variables, and secrets backends are three answers to one question: where does configuration live so that it isn’t in your code? A Connection names a system you reach; a Variable names a knob you turn; a secrets backend decides where either can be resolved from, ahead of Airflow’s own database, in a fixed order you can reason about. Get the split right and your DAGs stop being environment-specific scripts and start being portable descriptions of work — the same file, running against a throwaway Postgres on your laptop and a production warehouse behind Vault, differing only in which secret store answers the lookup.

That’s the end of Airflow from Scratch. You can stand up a local stack, write TaskFlow DAGs, schedule and backfill them, reach real systems through hooks, template the parts that vary, and now keep every credential out of git. Where the foundations stop, the Airflow in Practice series picks up — deferrable operators, custom XCom backends, executors, Kubernetes, and the deeper secrets-backend patterns this chapter only introduced.

Comments