Secrets Backends in Depth

A production treatment of Vault, AWS, and GCP secrets patterns, lookup order, prefixes, rotation, and why UI-stored secrets rarely scale.

The foundations series taught that secrets backends exist: point AIRFLOW__SECRETS__BACKEND at Vault or Secrets Manager, keep naming connections by id, and the credential stops living in the metadata database. That’s the intro. This chapter is the operating model — the part you need when there are three backends in play, when a secret rotated at 2am and half your workers still hold the old one, and when the bill from your secrets provider shows up itemized by API call because somebody put a Variable.get at module scope.

The mental model is unchanged. What changes is that every convenience in the intro has a production edge, and this is where we grind those edges down: exactly how a lookup name becomes a backend path, how the three big providers authenticate without shipping a key into the image, how to stack backends, how caching interacts with rotation, and how to prove a connection resolves before a DAG ever depends on it.

The full config, and the cache that sits under it

Two settings drive everything: the backend class and its keyword arguments, the latter as a JSON string.

export AIRFLOW__SECRETS__BACKEND='airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend'
export AIRFLOW__SECRETS__BACKEND_KWARGS='{"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables", "region_name": "us-east-1"}'

backend_kwargs being JSON matters more than it looks. It gets parsed once and passed as **kwargs to the backend’s __init__, so every value is a JSON scalar or object — a null is a real None, a number is a number, and a nested extra-style object stays an object. Malformed JSON here fails the whole secrets subsystem at startup, which is the good failure: you find out at boot, not on the first task that needs a password.

The resolution order is the fact that explains every “why is it using the wrong credential” incident:

  1. Environment variables — AIRFLOW_CONN_* / AIRFLOW_VAR_*
  2. The configured secrets backend — Vault, Secrets Manager, GCP, custom
  3. The metadata database — UI rows, airflow_settings.yaml

Airflow returns the first hit and stops. Environment always wins, which is a deliberate override hatch and a silent trap in equal measure — a stale AIRFLOW_CONN_BOOKSHOP_DB on one worker will shadow the correct value in the backend and never show up in the UI.

What the foundations chapter didn’t cover is the secrets cache that sits underneath this order. DAG parsing hits connections and variables constantly, and in 3.x you can let Airflow cache those lookups so the scheduler isn’t re-fetching the same secret on every parse:

export AIRFLOW__SECRETS__USE_CACHE=True
export AIRFLOW__SECRETS__CACHE_TTL_SECONDS=900   # 15 minutes

This cache is scoped to a process and is used during parsing, not task execution — a running task still resolves fresh. But the TTL is now part of your rotation math: a secret you rotate is guaranteed to be picked up by parsing only after the cache expires, up to cache_ttl_seconds later. Turn the cache on to protect your backend’s rate limits and bill; know that you’ve traded a slice of rotation latency for it. Leave it off (the default) and every parse pays the backend round-trip, which on a busy scheduler with a metered provider is a line item you’ll notice.

One more property of that cache decides how you reason about it: it lives inside each parsing process, not in a shared store, so the scheduler, the DAG processor, and each worker keep their own copy with independent TTLs. A rotation doesn’t propagate to the whole fleet at one instant — it lands process-by-process as each one’s TTL lapses. Size the TTL for the slowest propagation you can tolerate, not the average, because the worst case is what a rotation runbook has to plan around.

How a name becomes a path

A backend exposes three namespaces — connections, variables, and Airflow config — each with its own prefix. The prefix plus a separator plus the id is the literal path the backend fetches:

connections_prefix + sep + conn_id   ->  airflow/connections/bookshop_db
variables_prefix   + sep + var_key   ->  airflow/variables/bookshop_batch_size
config_prefix      + sep + config    ->  airflow/config/sql_alchemy_conn

config_prefix is the one the intro skipped, and it’s genuinely useful: it lets Airflow resolve its own sensitive settings — the metadata DB connection string, the broker URL, the Fernet key itself — from the same backend, instead of baking them into env vars on the box. Set config_prefix and store a secret at airflow/config/sql_alchemy_conn, and Airflow will pull its database URL from Vault at startup. That closes the last gap where a real credential still lived in plaintext on the host.

The separator is provider-specific and it’s the detail that bites people porting config between clouds. AWS and Vault use / and their paths look like directories. GCP Secret Manager forbids / in secret names — names are [A-Za-z0-9_-] only — so its backend defaults sep to -, and the same connection resolves to airflow-connections-bookshop_db. Same id, same prefix intent, completely different literal name. Get this wrong and every lookup misses.

Each prefix can be disabled independently by setting it to null:

{"connections_prefix": "airflow/connections", "variables_prefix": null}

That tells Airflow: resolve connections from this backend, but never search it for variables — skip straight to the metadata database for those. This is not just tidiness. Every lookup that isn’t disabled is a backend round-trip even on a miss, and a miss still costs a call and its latency. If you keep connections in Vault but variables in the database, disabling variables_prefix removes an entire class of wasted, billable, latency-adding calls that would otherwise fire on every variable read and fail. Disable what a backend doesn’t hold.

There’s a sharper version of the same lever. Every backend also takes a connections_lookup_pattern and variables_lookup_pattern — a regex the id must match before the backend is consulted at all:

{"connections_prefix": "airflow/connections", "connections_lookup_pattern": "^bookshop_"}

That tells the backend to answer only for connections whose id starts with bookshop_ and fall straight through for everything else, with no round-trip. Where null disables a whole namespace, the pattern disables it selectively: keep the fleet’s shared warehouse_* connections in the metastore while routing only your team’s bookshop_* ids to Vault. It’s the cleanest way to scope a backend to exactly the ids it owns, and on a metered provider it’s the same miss-cost saving as null, applied per-id instead of per-namespace.

HashiCorp Vault, three ways to authenticate

Vault is the richest of the three because Airflow has to become something Vault trusts before it can read anything, and the auth method is where the real production decisions live. The class is VaultBackend from the HashiCorp provider.

The shape:

export AIRFLOW__SECRETS__BACKEND='airflow.providers.hashicorp.secrets.vault.VaultBackend'
export AIRFLOW__SECRETS__BACKEND_KWARGS='{
  "connections_path": "connections",
  "variables_path": "variables",
  "config_path": "config",
  "mount_point": "airflow",
  "kv_engine_version": 2,
  "url": "https://vault.internal:8200",
  "auth_type": "approle",
  "role_id": "9d1e...",
  "secret_id": "b47c..."
}'

Two Vault-specific kwargs need naming. mount_point is the KV secrets engine’s mount — Vault lets you mount the same engine type at many paths, and airflow here means a bookshop_db connection lives at airflow/connections/bookshop_db in Vault’s addressing. kv_engine_version is a genuine footgun: KV v2 (the default for new Vault mounts) versions secrets and stores data under a data/ sub-path internally, while v1 does not. Set this to the wrong number and every read fails with a confusing 404, because the backend is looking at the wrong physical path. Check vault secrets list -detailed and match it.

Now the three auth methods, which is where team maturity shows:

Token — the simplest and the one you should outgrow. You hand Vault a token directly:

{"auth_type": "token", "token": "hvs.CAESI...", "url": "https://vault.internal:8200", "mount_point": "airflow"}

Fine for a spike. In production it’s a long-lived credential sitting in an env var — the exact thing secrets backends exist to eliminate — so treat it as a stepping stone, not a destination.

AppRole — the standard for a service that runs outside Kubernetes. Vault issues a role_id (stable, non-secret-ish, like a username) and a secret_id (rotating, like a password), and Airflow exchanges the pair for a short-lived token it refreshes itself. The role_id/secret_id in the config block above is this method. The win is that the secret_id can be delivered by a Vault agent or CI at deploy time and rotated on its own schedule, so nothing durable lives in the image.

Kubernetes — the cleanest when Airflow runs in a cluster, because it uses no static credential at all. Vault trusts the pod’s projected service-account JWT, and Airflow presents that token:

{
  "auth_type": "kubernetes",
  "kubernetes_role": "airflow-worker",
  "kubernetes_jwt_path": "/var/run/secrets/kubernetes.io/serviceaccount/token",
  "url": "https://vault.internal:8200",
  "mount_point": "airflow",
  "kv_engine_version": 2
}

kubernetes_role is the Vault role bound to the worker’s service account; kubernetes_jwt_path is where the kubelet projects the pod token. Nothing secret is in the config — the pod’s identity is the credential, and it’s short-lived and rotated by the platform. On Kubernetes this is the method to reach for; it’s the same argument as IAM roles on AWS, applied to Vault.

Two more Vault kwargs matter once you’re past a single team. Vault Enterprise partitions everything into namespaces, and if your platform runs Airflow under one, "namespace": "engineering/data" is mandatory — omit it and every read 404s against the root namespace, which looks identical to a wrong kv_engine_version and burns an afternoon of debugging the wrong thing. And the token Airflow receives from AppRole or Kubernetes auth is short-lived by design: the backend holds an hvac client per process and re-authenticates when the lease expires, so you set the token TTL on the Vault role, not anywhere in Airflow. Keep it long enough to cover a parse cycle and short enough that a leaked token dies on its own — minutes to an hour is the usual band. The whole appeal of AppRole and Kubernetes auth is that this expiry is automatic; a token credential has none, which is the other reason to outgrow it.

AWS Secrets Manager, on the instance’s identity

The Amazon provider’s SecretsManagerBackend is the least ceremony of the three because AWS’s credential chain does the authentication for you.

export AIRFLOW__SECRETS__BACKEND='airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend'
export AIRFLOW__SECRETS__BACKEND_KWARGS='{
  "connections_prefix": "airflow/connections",
  "variables_prefix": "airflow/variables",
  "config_prefix": "airflow/config",
  "region_name": "us-east-1"
}'

There is no key or token in that config, and that’s the whole point. The backend uses boto3’s default credential chain, so on a worker it authenticates as the IAM role attached to the task — an ECS task role, an EC2 instance profile, or an IRSA-bound service account on EKS. You attach a policy that grants secretsmanager:GetSecretValue scoped to airflow/*, and the worker can read exactly its own secrets and nothing else. If you must cross accounts you can add "role_arn" to have the backend assume a role, and "profile_name" uses a named local profile for dev — but the production default is “the machine’s own identity, least-privileged.”

Two AWS-specific knobs earn their keep. region_name should be set explicitly; leaving it to the environment is how a multi-region deployment ends up reading secrets from the wrong copy. And full_url_mode controls how a connection secret is stored: true (the default) means the secret string is a connection URI, while false lets you store a JSON object of connection fields — the readable form that dodges URI percent-encoding for passwords full of @ and /. Pick one convention and hold it, because the two aren’t interchangeable per-secret without the reader knowing which to expect.

Creating the secrets is the URI or JSON you already know:

aws secretsmanager create-secret \
  --name airflow/connections/bookshop_db \
  --secret-string 'postgresql://bookshop:[email protected]:5432/orders?sslmode=require'

aws secretsmanager create-secret \
  --name airflow/variables/orders_api_key \
  --secret-string 'sk_live_9f2a...'

With full_url_mode set to false, that first secret becomes the readable JSON form instead — which is what you want the moment a password contains an @, /, or % that URI-encoding would otherwise mangle:

aws secretsmanager create-secret \
  --name airflow/connections/bookshop_db \
  --secret-string '{"conn_type": "postgres", "host": "db.internal", "port": 5432, "login": "bookshop", "password": "s3c/r3t@!", "schema": "orders", "extra": {"sslmode": "require"}}'

The keys are the connection fields Airflow already knows — conn_type is required, and extra stays a nested object rather than a URL-encoded query string, so no character in password needs escaping. Choose full_url_mode: false fleet-wide before you store the first secret; flipping it later means every secret written under the old convention is now read with the wrong parser, and there’s no per-secret signal telling the backend which is which.

GCP Secret Manager, and the hyphen

The Google provider’s CloudSecretManagerBackend mirrors AWS’s model — authenticate as the workload, not as a stored key — with the naming twist from earlier.

export AIRFLOW__SECRETS__BACKEND='airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend'
export AIRFLOW__SECRETS__BACKEND_KWARGS='{
  "connections_prefix": "airflow-connections",
  "variables_prefix": "airflow-variables",
  "config_prefix": "airflow-config",
  "sep": "-",
  "project_id": "bookshop-prod"
}'

Note the prefixes use - and sep is -, because a GCP secret named airflow/connections/bookshop_db is illegal — the resolved name is airflow-connections-bookshop_db, and that’s what you must actually create:

echo -n 'postgresql://bookshop:[email protected]:5432/orders?sslmode=require' | \
  gcloud secrets create airflow-connections-bookshop_db \
    --project bookshop-prod --data-file=-

Authentication follows Application Default Credentials. On GKE with Workload Identity, or on any GCE-family compute, the attached service account is the credential and the config carries nothing sensitive — same pattern as AWS IAM and Vault’s Kubernetes auth, third verse. For off-GCP or local use you can point at a key file with "gcp_key_path": "/secrets/sa.json", but reaching for that in production usually means you’ve skipped a Workload Identity binding that would have let you carry no key at all. Grant the service account roles/secretmanager.secretAccessor scoped to the airflow-* secrets and you have the same least-privilege story as the other two.

Env vars, multiple backends, and rolling your own

The AIRFLOW_CONN_* / AIRFLOW_VAR_* convention isn’t just a dev convenience — because it sits above the backend in resolution order, it’s the right tool for a specific production job: a per-worker or per-environment override that must win unconditionally. A canary pool that should hit a read-replica, a break-glass credential injected for one deploy, a CI run pinned to a mock endpoint — all of these are AIRFLOW_CONN_* exports precisely because they beat whatever the shared backend says, with no backend change and no database row. The rule of thumb: the backend holds the default truth for the fleet; an env var expresses a local exception to it.

When one backend can’t hold everything — you keep application connections in Vault but the platform team publishes shared variables in AWS — Airflow does not natively chain two secrets backends. The clean way to get multi-backend behavior is to write a thin backend that delegates. Subclassing BaseSecretsBackend is a smaller job than it sounds; you implement the lookups you care about and return None to fall through to the next stage of the resolution order:

from airflow.secrets import BaseSecretsBackend
from airflow.providers.hashicorp.secrets.vault import VaultBackend
from airflow.providers.amazon.aws.secrets.secrets_manager import SecretsManagerBackend


class BookshopSecrets(BaseSecretsBackend):
    """Connections from Vault, variables from AWS Secrets Manager."""

    def __init__(self, **kwargs):
        super().__init__()
        self._vault = VaultBackend(mount_point="airflow", connections_path="connections",
                                   variables_path=None, url=kwargs["vault_url"],
                                   auth_type="kubernetes", kubernetes_role="airflow-worker")
        self._aws = SecretsManagerBackend(variables_prefix="airflow/variables",
                                          connections_prefix=None, region_name="us-east-1")

    def get_conn_value(self, conn_id: str):
        return self._vault.get_conn_value(conn_id)      # None -> Airflow tries the metastore next

    def get_variable(self, key: str):
        return self._aws.get_variable(key)

The methods that matter are get_conn_value(conn_id) (return a serialized connection — a URI or JSON string — or None), get_variable(key), and get_config(key). Returning None from any of them is not an error; it’s how you say “not here, keep looking,” and Airflow proceeds to the next source in the order. Point AIRFLOW__SECRETS__BACKEND at your class and pass vault_url and friends through backend_kwargs. The same hook is how you’d wrap a provider that doesn’t ship an Airflow backend at all — a secrets API internal to your company becomes a first-class Airflow source in about thirty lines.

Rotation, and the Fernet key nobody rotates

Rotation is the real test of the whole setup, and the promise is simple: rotate a database password in the backend, and workers pick up the new value without a DAG edit or a redeploy, because every task resolves the connection fresh at run time. That promise holds only if nobody defeated it — a top-level Variable.get or BaseHook.get_connection at module scope caches the value into a long-lived parsing process, and now a rotation doesn’t take effect until the next parse (and, if the secrets cache is on, not even then until the TTL lapses). The discipline from the foundations chapter — resolve inside tasks and templates, never at module scope — is what makes rotation work, not just what keeps the scheduler fast.

The credential everyone forgets to rotate is the Fernet key. It encrypts sensitive fields at rest in the metadata database, and once a secrets backend holds your real credentials, Fernet is protecting less than it used to — but it still guards anything that does land in the metastore, and losing it turns those rows into unrecoverable ciphertext. Airflow supports rotating it without downtime: set the new key first with the old one kept behind it as a comma-separated fallback, run the rotation command, then drop the old key.

export AIRFLOW__CORE__FERNET_KEY='<new-key>,<old-key>'   # new key first, old key still valid
airflow rotate-fernet-key                                 # re-encrypts every row with the new key
export AIRFLOW__CORE__FERNET_KEY='<new-key>'              # retire the old key

The order is load-bearing: new key leading, old key trailing, so both decrypt during the window; once rotate-fernet-key has re-encrypted everything, the old key is dead weight and comes out.

Cost, latency, and proving it resolves

Every backend lookup is a network round-trip and, on a metered provider, a billed call. The two forces that turn this from a footnote into an incident are both about frequency: a secret read at module scope multiplies by the parse rate (hundreds a minute across a fleet), and a non-disabled prefix pays a call even on a miss. The levers are the ones already introduced, used deliberately — resolve at task time so lookups fire only when work runs, null out prefixes a backend doesn’t hold so misses don’t call out, and turn on use_cache with a TTL you’ve reconciled against your rotation SLA. On a heavy scheduler against AWS Secrets Manager, those three together are frequently the difference between a trivial bill and a surprising one.

For local development the honest tool is LocalFilesystemBackend, which reads connections and variables from YAML/JSON files so your resolution path is byte-for-byte identical to production while the storage is a file on disk:

export AIRFLOW__SECRETS__BACKEND='airflow.secrets.local_filesystem.LocalFilesystemBackend'
export AIRFLOW__SECRETS__BACKEND_KWARGS='{"connections_file_path": "secrets/connections.yaml", "variables_file_path": "secrets/variables.yaml"}'

Those files hold plaintext and never enter git — the same rule as airflow_settings.yaml.

Finally, prove a connection resolves before a DAG depends on it. The CLI walks the exact resolution order Airflow uses at run time, so it’s the ground truth for “where is this actually coming from”:

airflow connections get bookshop_db

If that returns the value you expect, the backend, prefixes, auth, and separator are all correct end to end. If it returns something surprising, the resolution order is your debugging script: check the environment first (a stray AIRFLOW_CONN_*), the backend second, the metastore last — in that order, because that’s the order Airflow searched. The variable equivalent is airflow variables get orders_api_key, and inside the Astro CLI stack you run either through astro dev bash so it executes against the same config the scheduler sees. A connection you’ve confirmed with connections get is a connection you can build on; one you’ve only assumed is a 2am incident waiting for its logical date.

Examples in this chapter are docs-checked against the series’ stated Airflow 3.3.0 version, but they were not executed in this repository.

Final thoughts

The intro treats a secrets backend as a place secrets go. Production treats it as an identity and access problem wearing a storage costume — which is why the hard parts of this chapter were never about getting a value and always about who Airflow becomes to read it (AppRole, IRSA, Workload Identity, a Kubernetes JWT), how a name turns into a path across providers that don’t agree on a separator, and what happens on the second read when a cache, a TTL, and a rotation all have opinions. The backend doesn’t remove the need for least privilege; it relocates it. The worker’s identity should read only the secrets its DAGs need — one broad role over every production password just rebuilds the original risk in a system with better branding. Get the identity, the prefixes, and the resolution order right, and the same DAG file resolves against a file on your laptop and a Vault behind Kubernetes auth in prod, differing only in which store answers — and you can prove which one did, on demand, with a single command.

Next: REST API, External Triggers, and Dynamic DAGs

Comments