Variables, Settings, and Profiles

Prefect variables, profile switching, config, PREFECT_API_URL, local versus Cloud, and why blocks are not variables.

The blocks chapter drew the line and promised to come back to the other side of it. A block is the Prefect answer to an Airflow connection — typed config plus a client, credentials and the code that uses them in one object. Airflow Variables are a different animal, and Prefect’s exact analogue for them is what this chapter is about. prefect.variables.Variable is the true Airflow-Variable equivalent: a small, mutable, UI-and-CLI-editable JSON value with a name and nothing else. No client, no credential handling, no typed schema — just a knob you can turn without a deploy.

And once we’ve untangled variables from blocks, there’s a second, quieter layer that every Airflow engineer has an intuition for but no Prefect vocabulary yet: the machinery that decides which Prefect the CLI and your workers are even talking to. In Airflow that’s airflow.cfg, environment variables, and AIRFLOW__CORE__… overrides. In Prefect it’s settings and profiles, and getting them wrong is the single most common reason a flow “runs fine locally” and vanishes into a different backend in production.

A Variable is a name and a JSON value

Here is the whole surface area:

from prefect.variables import Variable

# write it once (REPL, setup script, CI, wherever)
Variable.set("bookshop_batch_size", 500)

# read it anywhere
size = Variable.get("bookshop_batch_size", default=100)

Variable.set(name, value) stores the value server-side, in the same Prefect API that holds your flow runs and blocks. Variable.get(name, default=...) reads it back. If the variable doesn’t exist and you passed default=, you get the default; if it doesn’t exist and you didn’t, you get None. That’s the Airflow Variable.get("key", default_var=100) pattern, beat for beat.

The values are JSON, not just strings — and this is the first place Prefect diverges usefully from Airflow habit. In Airflow, Variable.get returns a string unless you pass deserialize_json=True and remember to have stored valid JSON. Prefect variables are JSON natively. You can store a number, a boolean, a list, or a nested object and get the same Python type back:

Variable.set("bookshop_batch_size", 500)                 # int
Variable.set("bookshop_notify", True)                    # bool
Variable.set("bookshop_skip_publishers", ["acme", "old-house"])   # list
Variable.set(
    "bookshop_thresholds",
    {"min_orders": 10, "max_late_days": 3},              # dict
)

th = Variable.get("bookshop_thresholds", default={})
if th["max_late_days"] > 5:
    ...

No deserialize_json flag, no ast.literal_eval, no wondering whether the string you got back is "500" or 500. What you set is what you get.

Two operational notes. First, set is create-or-fail by default; to change an existing variable in code you pass overwrite=True:

Variable.set("bookshop_batch_size", 1000, overwrite=True)

Second, there’s a full async twin. Inside an async flow or task you don’t want a synchronous network call blocking the event loop, so use aget / aset:

from prefect import flow
from prefect.variables import Variable

@flow
async def nightly_load():
    size = await Variable.aget("bookshop_batch_size", default=100)
    skip = await Variable.aget("bookshop_skip_publishers", default=[])
    ...

The sync get/set and the async aget/aset are the same variable — the split is purely about which calling context you’re in, exactly the way the rest of Prefect 3.x gives you a sync and an a-prefixed async form of everything.

One timing subtlety worth stating plainly, because it’s the whole reason variables are useful for operations: Variable.get reads over the network at the moment it runs, not when the module imports. Call it at the top of a flow and each run picks up whatever the value is right then. That means an edit in the UI takes effect on the next run, and it also means you should read a variable once at the start of a flow and pass it down, rather than calling Variable.get in a tight loop — each call is an API round-trip, and a variable that changes mid-run would give different tasks different answers. Treat it as configuration you snapshot at the top of a run, not a live feed you poll.

From the CLI and the UI

Because a variable is just a named value on the API, you never need Python to manage one. The prefect variable command group covers it:

prefect variable set bookshop_batch_size 500
prefect variable set bookshop_thresholds '{"min_orders": 10, "max_late_days": 3}'
prefect variable get bookshop_batch_size
prefect variable ls
prefect variable inspect bookshop_thresholds
prefect variable unset bookshop_batch_size

set from the CLI takes the value as a string and stores it as JSON where it parses as JSON. The same variables appear in the Prefect UI under Variables, editable in a text box. This is the operational point: an on-call engineer can bump bookshop_batch_size from 500 to 100 during an incident, in the UI, with no deploy and no code change — the next flow run reads the new value the instant it calls Variable.get. That’s the same “config that changes on a different schedule than code” win blocks give you, applied to the small stuff that doesn’t deserve a typed class.

Variables in prefect.yaml

Variables also resolve inside your deployment definition. Anywhere in prefect.yaml you can template a variable with the {{ prefect.variables.<name> }} syntax, and Prefect substitutes its value when the deployment is built or when a run is scheduled:

deployments:
  - name: nightly-load
    entrypoint: flows/load.py:nightly_load
    work_pool:
      name: bookshop-pool
    parameters:
      batch_size: "{{ prefect.variables.bookshop_batch_size }}"
    tags:
      - "{{ prefect.variables.bookshop_env }}"

This is the mirror image of Airflow’s {{ var.value.bookshop_batch_size }} in a templated field — same idea, resolved at the deployment layer rather than in a Jinja-rendered operator argument. It lets one prefect.yaml, checked into git unchanged, produce differently-parameterized deployments depending on what the target environment’s variables hold. Hold that thought; it’s the backbone of the dev-vs-prod setup at the end of this chapter.

Variable, block, or parameter?

Now the record-straightening. You have three places to put a value that isn’t hard-coded, and they are not interchangeable. Choosing wrong is how projects get hard to operate.

  • A parameter is a per-run input. It changes every time you call the flow, or every time a schedule fires with a different argument. batch_size passed to nightly_load(batch_size=500) is a parameter. If the question is “what did this run operate on,” it’s a parameter. Parameters live in the flow signature and the run’s context; they are not stored anywhere between runs.

  • A variable is small, shared, mutable config — the same value across many runs, changed occasionally by a human, holding no secret. Batch sizes, feature flags, a list of publishers to skip, an environment name, a Slack channel to notify. If the question is “what setting is this environment currently using,” it’s a variable. It’s stored on the API, editable in the UI, and read fresh at run time.

  • A block is a typed external resource or a secret — a database connector, an S3 bucket handle, a warehouse credential. It carries a schema and often a live client. If the value is a password, an API key, or a connection to a system, it’s a block, not a variable. Chapter 7 is the full story.

If you’re porting an Airflow project, the sorting is usually quick. Anything you stored as an Airflow Variable — a JSON blob of feature flags, a list of accounts to process, a threshold — moves to a Prefect variable almost verbatim, and the code shrinks because you drop the deserialize_json=True and the string-parsing. Anything you stored as an Airflow Connection — a database DSN, an AWS credential, an HTTP endpoint with auth — moves to a block, gaining a typed schema and often a client. And anything you passed through dag_run.conf or an operator argument that varied per run stays a parameter. The one migration that bites is the Airflow Variable that was quietly holding a secret because someone didn’t want to make a Connection: that does not become a Prefect variable, it becomes a Secret block, because a Prefect variable is as visible as an Airflow Variable was and you don’t want the token in the clear.

The bright line between a variable and a block is secrets and types. Variables are stored in plaintext and shown in the UI in the clear — putting a password in a variable is exactly the mistake of putting a password in an Airflow Variable instead of a Connection. If it’s sensitive, it belongs in a Secret block. And if it needs structure and behavior — “this is a Snowflake connector with an account, a warehouse, and a .get_connection() method” — that’s a block’s typed schema, not a variable’s loose JSON. Variables are for the dumb little knobs; blocks are for the smart configured things. They’re two tools, and the reason the blocks chapter spent a table separating them is that Airflow’s habit of saying “connections and variables” in one breath makes them feel like one. Use variables for the values you’d have reached for an Airflow Variable to hold, and blocks for the ones you’d have reached for a Connection.

Settings: which Prefect are you talking to?

Every prefect command and every running flow needs to know a handful of things before it can do anything: what API to hit, what key to authenticate with, how verbose to log, where to write results. These are settings, and in Prefect 3.x they’re the direct descendant of Airflow’s airflow.cfg plus AIRFLOW__SECTION__KEY environment overrides.

Every setting has a canonical environment-variable name prefixed PREFECT_. The two that matter most are the ones that pick your backend:

PREFECT_API_URL     # the Prefect API this CLI / worker talks to
PREFECT_API_KEY     # the auth key (Prefect Cloud, or a secured self-hosted server)

PREFECT_API_URL is the whole ballgame for “why can’t I see my run.” Point it at http://127.0.0.1:4200/api and you’re talking to a local server you started with prefect server start. Point it at a Prefect Cloud workspace URL and you’re talking to Cloud. A flow submitted against one is completely invisible from the other — there is no shared state, just two different APIs. Most “my deployment disappeared” confusion is a CLI configured for one backend and a worker configured for another.

If you’re coming from Airflow, the mental map is almost mechanical. Airflow’s AIRFLOW__CORE__PARALLELISM becomes Prefect’s PREFECT_…-prefixed name; Airflow’s airflow.cfg sections become Prefect’s prefect.toml tables; Airflow’s per-environment env-var overrides become Prefect’s per-profile settings. The values differ, but the shape — a layered config where environment variables override a file which overrides defaults — is the same layered config you already reason about when you debug an Airflow deployment. A few settings beyond the API URL that come up constantly:

PREFECT_API_URL          # the API to talk to (the big one)
PREFECT_API_KEY          # auth for Cloud or a secured server
PREFECT_LOGGING_LEVEL    # DEBUG when a run is misbehaving
PREFECT_RESULTS_PERSIST_BY_DEFAULT   # persist task results without per-task config
PREFECT_HOME             # where ~/.prefect lives (profiles, local DB)

You inspect and change settings with the prefect config command group:

prefect config view              # show effective settings + where each came from
prefect config set PREFECT_API_URL=http://127.0.0.1:4200/api
prefect config set PREFECT_LOGGING_LEVEL=DEBUG
prefect config unset PREFECT_LOGGING_LEVEL

prefect config view is the command to reach for first when anything behaves unexpectedly. It prints the effective value of every setting and its source — whether it came from an environment variable, the active profile, or a default. It’s airflow config list with provenance attached, and it turns “which config won” from a guess into a fact.

Settings can be supplied from several places, and they resolve in a strict precedence order, highest wins:

  1. Environment variables — PREFECT_API_URL=… in the process environment. Always wins.
  2. A .env file in the working directory — the same PREFECT_* names, loaded automatically.
  3. prefect.toml (or [tool.prefect] in pyproject.toml) in the working directory — project-scoped settings checked into the repo.
  4. The active profile in ~/.prefect/profiles.toml — your machine’s current default.
  5. Built-in defaults.

The rule to internalize: environment beats profile beats default. A PREFECT_API_URL exported in a container’s environment overrides whatever profile that machine happens to have active, which is exactly what you want in production — the deployment’s environment is authoritative, and no developer’s leftover prefect profile use can redirect a production worker. Conversely, on your laptop the profile is the convenient default and you rarely set env vars by hand. Both mechanisms exist so the same settings can be chosen interactively during development and pinned explicitly in production.

prefect.toml deserves a word because it’s the newest and most Airflow-like of these. It’s a file you commit to the repo, next to prefect.yaml, holding settings that should travel with the project rather than the machine:

# prefect.toml
[logging]
level = "INFO"

[results]
persist_by_default = true

Mind the table name, because this one is a genuinely nasty near-miss. [logging] level is the setting you just met — the log level of your flows and tasks. There is also a [server] logging_level, and it looks like the obvious place to put it, and it is not: that one sets the log level of the server process itself. Put your INFO in [server] and Settings().logging.level doesn’t move at all — your flows keep logging at whatever they were, and you’ve configured the wrong program.

Anyone who checks out the repo and runs a flow from that directory inherits those settings, no per-machine setup required — the project-scoped analogue to shared airflow.cfg defaults, but scoped to a directory and overridable by environment above it.

Profiles: named bundles of settings

A profile is a named collection of settings stored in ~/.prefect/profiles.toml. One profile is active at a time, and switching profiles swaps every setting in it at once. This is the piece Airflow has no clean equivalent for — Airflow assumes one configured environment per machine; Prefect assumes you’ll routinely juggle several.

prefect profile ls               # list profiles, mark the active one
prefect profile create local     # make a new empty profile
prefect profile use local        # switch the active profile
prefect profile inspect local    # dump this profile's settings

The file itself is readable and hand-editable:

# ~/.prefect/profiles.toml
active = "local"

[profiles.local]
PREFECT_API_URL = "http://127.0.0.1:4200/api"

[profiles.staging]
PREFECT_API_URL = "https://api.prefect.cloud/api/accounts/<acct>/workspaces/<staging-ws>"
PREFECT_API_KEY = "pnu_staging_…"

[profiles.prod]
PREFECT_API_URL = "https://api.prefect.cloud/api/accounts/<acct>/workspaces/<prod-ws>"
PREFECT_API_KEY = "pnu_prod_…"

prefect config set writes into the active profile, so the CLI and the file are two views of the same thing. And when you run prefect cloud login, Prefect fills in PREFECT_API_URL and PREFECT_API_KEY for your chosen Cloud workspace into the active profile for you — the guided path in and out of Cloud without editing TOML by hand.

In practice you end up with a profile per backend, because Prefect assumes three of them and the profile is how you keep them straight:

  • A local server you start yourself with prefect server start — an ephemeral SQLite-backed API at http://127.0.0.1:4200/api, perfect for iterating on a laptop with no network dependency.
  • A self-hosted server your team runs — the same open-source Prefect server, but on shared infrastructure with a real database, reached over the network and (ideally) behind auth, so PREFECT_API_KEY comes into play alongside the URL.
  • Prefect Cloud workspaces — the hosted control plane, each workspace a distinct PREFECT_API_URL under api.prefect.cloud, always keyed.

A profile bundles the URL (and key, where needed) for each, so switching among all three is a one-liner:

prefect profile use local        # ephemeral local server
prefect profile use team          # shared self-hosted server
prefect profile use prod          # Prefect Cloud prod workspace

Every prefect deploy, prefect run, prefect variable, and prefect block you type afterward targets whatever the active profile points at. This is why prefect config view earns its keep: it always shows you which profile is active and therefore which backend you’re about to affect. The failure mode profiles exist to prevent is the subtle one — creating a variable or a block against local, then wondering why the Cloud worker can’t find it. It can’t find it because it was never there; you wrote it to a different API. Profiles don’t make that impossible, but they make the active backend a thing you name out loud rather than a thing you forget.

A worked dev-vs-prod setup

Let’s assemble the pieces into the setup a real bookshop project would run, where the same flow code reads a different batch size in development than in production, and where that difference lives in the environment, not in the code.

The flow reads its knob from a variable, with a safe default:

# flows/load.py
from prefect import flow, get_run_logger
from prefect.variables import Variable

@flow
def nightly_load():
    logger = get_run_logger()
    env = Variable.get("bookshop_env", default="dev")
    batch_size = Variable.get("bookshop_batch_size", default=100)
    logger.info(f"running in {env} with batch_size={batch_size}")
    # ... extract, transform, load in batches of batch_size ...

Nothing in this file knows or cares which environment it’s in. It asks the API for bookshop_env and bookshop_batch_size at run time and adapts. That’s the goal: one artifact, environment-dependent behavior, resolved at run time.

Now the two environments. On the laptop, a dev profile pointing at a local server, with small, cheap variable values:

prefect profile create dev
prefect profile use dev
prefect config set PREFECT_API_URL=http://127.0.0.1:4200/api

# seed dev's variables against the dev backend
prefect variable set bookshop_env dev
prefect variable set bookshop_batch_size 50

And the prod profile pointing at Cloud, with production-scale values seeded against the Cloud API:

prefect profile create prod
prefect profile use prod
prefect cloud login                 # fills in PREFECT_API_URL + PREFECT_API_KEY

# seed prod's variables against the Cloud backend
prefect variable set bookshop_env prod
prefect variable set bookshop_batch_size 2000

The crucial and easily-missed detail: variables live in the backend, so bookshop_batch_size is genuinely two different values in two different APIs. When the dev profile is active you’re reading and writing the local server’s copy (50); when prod is active you’re reading Cloud’s copy (2000). The variable name is shared; the value is per-backend. There’s no if env == "prod" branch anywhere in the flow — the branch is data, stored where the run happens.

Production doesn’t rely on anyone having run prefect profile use prod, though. The worker that actually executes runs is configured by environment, because environment beats profile:

# in the prod worker's container / systemd unit / k8s manifest
export PREFECT_API_URL="https://api.prefect.cloud/api/accounts/<acct>/workspaces/<prod-ws>"
export PREFECT_API_KEY="pnu_prod_…"
prefect worker start --pool bookshop-pool

Those exported PREFECT_* values override any profile on the box, so the worker is nailed to the prod workspace no matter what. The profile is the human’s convenient default; the environment variable is the machine’s pinned truth. Same settings system, two mechanisms, chosen deliberately per context.

Finally, prefect.yaml ties parameters to variables so even the deployment’s inputs follow the environment:

deployments:
  - name: nightly-load
    entrypoint: flows/load.py:nightly_load
    work_pool:
      name: bookshop-pool
    parameters:
      batch_size: "{{ prefect.variables.bookshop_batch_size }}"

Deploy this against the dev backend and the scheduled runs carry batch_size=50; deploy the identical file against Cloud and they carry 2000. When something looks wrong in either place, the first command is always the same:

prefect config view          # which profile? which API? where did each setting come from?

If the answer is “the wrong workspace,” you’ve found your bug before reading a single line of flow code.

Final thoughts

The clean version of the rule, now that variables and blocks are properly separated:

  • Parameters are per-run inputs — they live in the flow signature and change every run.
  • Variables are small, shared, mutable, non-secret config — plain JSON knobs on the API, editable in the UI, read fresh at run time. This is the true Airflow-Variable analogue — the other half of the block-versus-variable line the blocks chapter drew.
  • Blocks are typed external resources and secrets — schemas and clients and credentials, never plaintext knobs.
  • Settings and profiles choose the control plane — which Prefect API, which key, which defaults — with environment beating profile beating default, so humans get convenient switching and machines get pinned truth.

Airflow taught you to keep config out of code; Prefect keeps the same discipline but hands you sharper instruments for it, and the whole system only stays legible if you use each one for the job it’s shaped for. Put a secret in a variable and you’ve leaked it; scatter if env == "prod" through your flows and you’ve hard-coded the thing that was supposed to be data; let a worker inherit whichever profile a developer last touched and you’ve made production non-deterministic. Get the four boxes right and an on-call engineer can retune a live pipeline from a text box, and a new hire can point their laptop at the local server, run the exact production flow, and never once edit its code.

Next: Pause, Resume, and Human-in-the-Loop Flows

Comments