Cloud or Self-Hosted: Who Gets Paged for the Orchestrator?

This isn't a hosting decision, it's an on-call decision — an unsentimental itemisation of what Prefect Cloud sells you, what self-hosting the server really costs, and the hybrid boundary almost nobody can draw correctly.

Every Prefect evaluation eventually arrives at a slide with two columns on it, and the slide is always wrong. On the left, “Self-hosted: free.” On the right, “Cloud: $$$.” Somebody says “we’ll self-host, it’s open source,” everyone nods, and eighteen months later a data engineer is awake at 3 a.m. because a Postgres connection pool exhausted itself and now no flow run in the company can transition out of Pending.

The column headers are the problem. You are not choosing a hosting option. You are choosing who operates the control plane — whose pager goes off when the thing that remembers what your pipelines are doing stops remembering, and whose weekend gets spent on the upgrade. Free is a price on a licence, not a price on a system. The self-hosted column has a number in it too; it’s just denominated in engineer-hours and written in invisible ink.

You have made this decision before, incidentally — Airflow forced it on you the first time someone said “MWAA or a cluster?” So you have priors, and most of them don’t transfer, because the thing being hosted is a different shape in both directions: Prefect’s open-source server is much lighter to run than Airflow’s, and it gives you much less than Airflow’s does. Both halves of that sentence matter, and the chapter is spent earning them.

What the control plane actually is

Be precise about what you’d be hosting, because the fuzziness is where the bad estimates come from. A Prefect control plane is four things:

  • An API — a FastAPI app. Every client you have talks to it: the CLI, your flows, your workers, CI.
  • A database — SQLite for a laptop, Postgres for anything real. This is the whole product. Every flow run, task run, log line, state transition, event, block document, variable, deployment, schedule, work pool, and automation lives here.
  • A UI — a static React app the API serves.
  • Loop services — the background processes that make things happen: the scheduler that creates runs from schedules, the one that marks runs Late, the trigger evaluator that fires your automations, the event persister that writes the event stream to disk.

And here is the crucial thing it is not: an executor. The control plane does not run your flows. Workers do, in your infrastructure, on both editions — that’s chapter 06’s whole argument. Which means the blast radius of the control plane going down is narrower than an Airflow engineer’s instinct expects. A flow run that is already executing when the API falls over doesn’t die; it keeps running and fails to report, because the code is in your pod, not on the server.

That’s the good news, and it is thinner than it sounds. Nothing new gets scheduled. Nothing gets triggered. No automation fires — so the 3 a.m. runbook from chapter 11, the one that pages a human when the nightly load hangs, is exactly as down as everything else. No block loads, so any flow that starts by calling SqlAlchemyConnector.load(...) fails on line one. No variable resolves. Your on-call engineer can’t see anything, because the UI is the API. The control plane isn’t in the data path, but it is in the control path, and “the orchestrator is fine, we just can’t orchestrate” is a distinction nobody will let you make on the incident call.

So: who keeps that up, and who is accountable when it isn’t?

Self-hosting, properly

prefect server start is a development toy. I want to say that early and without hedging, because it is a fantastic toy — one command, a UI on 4200, zero configuration — and that experience is precisely what convinces teams the production version is also one command. It is not. Here is the gap, item by item.

Postgres, and the URL you’ll get wrong once

SQLite is the default and it is disqualifying for production. Not “suboptimal” — disqualifying. Prefect’s API is an async web server with multiple workers and a fleet of loop services all writing concurrently, and SQLite’s writer lock will turn that into database is locked under exactly the load you built the thing for.

Point it at Postgres before you start it:

prefect config set PREFECT_SERVER_DATABASE_CONNECTION_URL=\
"postgresql+asyncpg://prefect:[email protected]:5432/prefect"

Two notes. The +asyncpg is not decoration — Prefect talks to Postgres over the async driver, and a postgresql:// or postgresql+psycopg2:// URL will fail. And if you’ve read older material (or chapter 02), you’ll have seen PREFECT_API_DATABASE_CONNECTION_URL; that name still works as a legacy alias, but PREFECT_SERVER_DATABASE_CONNECTION_URL is the current one and it’s what the settings model calls canonical.

In a container you almost certainly don’t want the connection URL, because it embeds the password in a single string that shows up in prefect config view, in your manifest, and in anything that dumps the environment. The database settings decompose, which is the shape you want when the password comes from a Kubernetes secret and nothing else does:

env:
  - name: PREFECT_SERVER_DATABASE_DRIVER
    value: "postgresql+asyncpg"
  - name: PREFECT_SERVER_DATABASE_HOST
    value: "postgres.internal"
  - name: PREFECT_SERVER_DATABASE_PORT
    value: "5432"
  - name: PREFECT_SERVER_DATABASE_NAME
    value: "prefect"
  - name: PREFECT_SERVER_DATABASE_USER
    value: "prefect"
  - name: PREFECT_SERVER_DATABASE_PASSWORD
    valueFrom:
      secretKeyRef: { name: prefect-db, key: password }

Same result, one secret, no credential smeared across a URL string.

Migrations are your upgrade path, and reset is not

Every Prefect release can carry Alembic migrations. Upgrading the server means running them, and the command is:

prefect server database upgrade -y

It takes --dry-run (prints the migrations without applying them — run this in CI against a restored snapshot before you ever run it against prod) and --revision (pin to a specific Alembic revision rather than head). There’s a matching prefect server database downgrade if a release goes badly, though as with every downgrade in every system, the migration that dropped a column is not coming back to help you.

The operational rule is boring and non-negotiable: migrations run before the new server serves traffic. In Kubernetes that’s an init container or a Job with a helm.sh/hook: pre-upgrade annotation; in a compose setup it’s a one-shot container that must exit 0 before the API container starts. If you skip this and roll a new image, you get an API talking to a schema it doesn’t recognise, and the errors will be creative.

And then there is the command that sits one keystroke away from all of this:

prefect server database reset      # DROPS AND RECREATES ALL TABLES

It is in chapter 02 as the friendly “unstick my laptop” move, and against ~/.prefect/prefect.db that’s exactly what it is. Against a production Postgres it deletes every flow run, block, variable, automation, and deployment your team has ever created — and it does it against whatever API your active profile currently points at, which, after chapter 15, you’ll recognise as a sentence that ends careers. The profile you forgot you switched is the profile it uses. If reset must appear in a runbook at all, put a mandatory prefect config view above it.

Auth: one password, and exactly what it buys

The folk wisdom is “the OSS server has no auth.” That’s wrong, and repeating it will make you either over-fear or under-defend. It has auth. It has one password.

Set it on the server:

PREFECT_SERVER_API_AUTH_STRING="admin:hunter2" prefect server start

Now the API is gated with HTTP Basic. Unauthenticated calls get a 401, the UI presents a login, and a wrong password gets a 401 too. And every client — every worker, every CI job, every developer’s laptop, every python bookshop.py — must present the same credential, under a different setting name that catches people out:

export PREFECT_API_URL="https://prefect.internal/api"
export PREFECT_API_AUTH_STRING="admin:hunter2"    # note: API, not SERVER_API
prefect work-pool ls

Miss that on a client and you don’t get a friendly message; you get PrefectHTTPStatusError: Client error '401 Unauthorized' for url '.../api/csrf-token?client=...', which sends people hunting for a CSRF bug that isn’t there.

One wrinkle worth knowing before it fools you: /api/health is not gated. It answers 200 to an unauthenticated request, by design, so your load balancer’s probe doesn’t need credentials. The trap is that “I curled the server and got a 200, so it’s up and it’s fine” tells you nothing about whether auth is on. Curl something real — POST /api/flow_runs/filter with an empty body — and see if you get a 401. If you get a 200, your orchestrator is on the open internet.

Now the honest accounting of what that password isn’t:

  • It is not identity. Every request looks identical to the server. “Who cancelled that flow run at 02:14?” has no answer, and never will, because the information was never collected.
  • It is not authorization. There are no roles. Everyone who can read the UI can delete every deployment in it. The read-only stakeholder who wants to check whether the nightly load finished has the same power as your platform lead.
  • It is not revocable per-actor. Rotating it means changing the string on the server and simultaneously in every worker, every CI secret, and every laptop. There is no “revoke Sarah’s access” — there is only “change the password for everybody, at once, and then go find out what broke.”

That’s a real security control and it is worth turning on — it is the difference between an exposed orchestrator and a defended one. It is just not a governance story, and if your compliance questionnaire has the word “least privilege” in it, this is where the conversation ends.

The reverse proxy, and the three settings that make the UI work

The server speaks plain HTTP and has no opinion about TLS. Production means a reverse proxy in front — nginx, an ALB, an ingress — terminating TLS and forwarding to :4200. Three configuration details will bite you in order:

Websockets. The event stream (chapter 11) and the UI’s live-updating run views are websockets. A proxy that doesn’t forward the upgrade handshake gives you a UI that loads, looks perfect, and never updates — and an engineer who spends a day convinced the scheduler is broken:

location / {
    proxy_pass         http://prefect-server:4200;
    proxy_http_version 1.1;
    proxy_set_header   Upgrade    $http_upgrade;
    proxy_set_header   Connection "upgrade";
    proxy_set_header   Host       $host;
    proxy_read_timeout 300s;
}

PREFECT_UI_API_URL. The UI is a browser app, and it calls the API from the browser, not from the server. Its default is http://127.0.0.1:4200/api, which is correct on a laptop and catastrophic behind a proxy: the browser dutifully tries to call the user’s own machine. Set it to the URL the outside world uses:

PREFECT_UI_API_URL="https://prefect.internal/api"

CSRF and CORS. PREFECT_SERVER_CSRF_PROTECTION_ENABLED defaults to False, and PREFECT_SERVER_CORS_ALLOWED_ORIGINS defaults to *. Both defaults are chosen for the laptop case and neither is what you want on a server that other humans reach:

PREFECT_SERVER_CSRF_PROTECTION_ENABLED=true
PREFECT_SERVER_CORS_ALLOWED_ORIGINS="https://prefect.internal"

None of this is exotic. All of it is yours, and all of it is work that a managed control plane simply doesn’t generate.

Event retention: the setting that quietly deletes your forensics

This one deserves its own heading because it is the single most expensive default in self-hosted Prefect, and you will discover it during a post-mortem or not at all.

PREFECT_SERVER_EVENTS_RETENTION_PERIOD    # default: 7 days

Seven days. After that, events are deleted. And your event stream is your forensic trail — it’s the record of every state transition, every deployment, every work-pool health change, and every domain event you emitted with emit_event(). It is also the substrate automations match against, which means a proactive trigger’s whole world is that window.

So when someone asks “this pipeline has been silently producing bad data since the last release — show me when the behaviour changed,” and the last release was eleven days ago, the answer on a default self-hosted server is: we don’t know, and we can’t find out. The data is gone. Not archived, not cold — gone.

Raise it deliberately, and size the disk to match:

prefect config set PREFECT_SERVER_EVENTS_RETENTION_PERIOD="90 days"

Events are high-cardinality and high-volume — every task run in every flow run emits several — so 90 days of retention on a busy deployment is a real Postgres table with real growth, and “grow the volume before it fills” is now a thing on your plate. That’s the deal. The default protects your disk at the cost of your memory, and nobody tells you which one you traded.

(While you’re in there: PREFECT_SERVER_SERVICES_DB_VACUUM_RETENTION_PERIOD defaults to 90 days and governs the periodic cleanup of other aged rows. Know it exists before your DBA asks why the database only ever grows.)

Splitting the API from the services

prefect server start runs the API and every loop service in one process. That’s convenient and it means an API restart takes your scheduler and your trigger evaluator down with it. You can split them:

prefect server start --no-services      # API + UI only; scale this behind the LB
prefect server services start           # the scheduler, trigger evaluator, persister, …
prefect server services ls              # what's enabled, and the setting that gates each

Now the API is a horizontally-scalable stateless web tier and the services are a single replica you can restart independently. prefect server services ls is worth running once just to see the list — TaskRunRecorder, EventPersister, ReactiveTriggers, Actions, Distributor and friends, each with the PREFECT_SERVER_SERVICES_* setting that turns it on or off. These are the moving parts you have adopted.

The Helm reality check

Prefect publishes a Helm repo with two charts. Everybody uses one of them.

helm repo add prefect https://prefecthq.github.io/prefect-helm
helm install prefect-worker prefect/prefect-worker ...   # extremely well-trodden
helm install prefect-server prefect/prefect-server ...   # comparatively lonely

The worker chart is the one chapter 06 showed, and it’s mature because it’s the path every Prefect Cloud customer on Kubernetes takes. The server chart exists and works, but the population running it in anger is a fraction of that, and it shows in the places you’d expect: fewer people have hit your ingress bug before you, fewer write-ups of your Postgres tuning, fewer Slack threads that end in an answer. Self-hosting the Prefect server is not exotic, but it is lonelier than self-hosting Airflow, where a decade of engineers have already stepped on every rake in the yard and blogged about it. Budget for being the one who finds the rake.

And back it up like it’s a database, because it is

Your Postgres holds every block (including credentials), every variable, every automation, every deployment and schedule, and all of your run history. Losing it does not stop the flow that’s running right now. It stops absolutely everything else, permanently. Back it up on the schedule you’d use for a production transactional database, and — this is the part teams skip — restore it somewhere, once, on purpose, so that the first time you find out your backup is a directory of zero-byte files is not the day you need it.

What Prefect Cloud actually sells you

Now the other column, itemised and unsentimental. The mistake here is assuming you’re paying for a scheduler you don’t have to run. You aren’t, particularly — the OSS scheduler is the same code and it’s fine. What you’re buying is identity and governance, plus a handful of features that genuinely cannot exist without a hosted, addressable, multi-tenant control plane.

Workspaces. A hard isolation boundary with its own API URL, its own blocks, its own deployments, its own run history. Dev, staging, and prod as three workspaces you switch between with prefect cloud workspace set --workspace "acme/prod". On OSS the equivalent is three separate servers and three separate Postgres instances that you run and upgrade three times.

Users, and RBAC. Real per-user identity, with roles scoped per workspace. This is the one OSS structurally cannot approximate — not “hasn’t got around to,” cannot, because a single shared password carries no identity to attach a role to. Every governance feature below is downstream of this one.

SSO, and SCIM provisioning. Access follows your identity provider. Someone joins the data team in Okta and gets a Prefect seat; someone leaves the company and loses it — without anyone remembering to go and do it. Deprovisioning that depends on human memory is deprovisioning that doesn’t happen.

Service accounts. Non-human identities with their own keys and their own roles. This is important enough that it gets its own section below.

Audit logs. “Who paused that deployment?” is a question with an answer. It’s a compliance checkbox, sure, but it’s also the thing that turns a confusing incident into a five-minute one.

Incidents. The declare-incident action from chapter 11, plus the tracking surface around it.

Inbound webhooks. prefect cloud webhook create — and that cloud prefix in the command is not an accident. A webhook needs a stable, externally-addressable, authenticated URL, which is a thing a hosted platform has and a server behind your VPN doesn’t.

Metric triggers. The successes / duration / lateness triggers are Cloud-only, and the OSS server doesn’t just ignore them — it doesn’t have the type. POST a metric trigger to a self-hosted /api/automations/ and you get a 422, while an event, compound, or sequence trigger POSTs happily and returns a 201. It’s an unhelpful way to learn a licensing boundary, so learn it here instead.

Longer event retention. The forensic window above, as a plan feature rather than a disk you have to grow.

Push and managed work pools. ecs:push, cloud-run:push, azure-container-instance:push, modal:push, and prefect:managed — the worker-free execution models from chapter 06. Cloud only, because it’s the Cloud API that calls your cloud provider’s API on your behalf.

IP allowlisting. prefect cloud ip-allowlist add restricts which addresses can reach your account at all — the answer to “but our orchestrator’s control plane is on the public internet.”

And the cost, honestly: it’s usage-based, and the line item that surprises people is not seats — it’s run volume. A team with six engineers and four hundred thousand task runs a month is not paying a six-seat bill. Model it on your actual task-run count before you commit, and remember that Prefect’s fan-out ergonomics (chapter 04) make it very easy to turn one Airflow task into two thousand Prefect task runs without noticing.

The hybrid boundary, drawn properly

This is the strongest thing Cloud has, and in my experience almost nobody who uses it can state it accurately — including, embarrassingly often, the person defending it to their security team.

Workers only ever make outbound calls. A worker polls the Prefect API over HTTPS, takes the runs it’s meant to execute, and reports states back. It does not listen on a port. Nothing — not Prefect, not anyone — connects inward to it. Which means you can run workers deep inside a private subnet with no ingress, no VPN, no public endpoint and no inbound firewall rule, and the whole system works, because the only network flow is your VPC dialling out on 443.

That’s the shape. Now the boundary, in both directions, without the marketing gloss.

What never crosses:

  • Your flow code. The worker pulls it — from your git remote, your registry, your S3 bucket. Cloud stores a pointer (an entrypoint and a storage reference), not a copy.
  • Your data. Rows never traverse Prefect. The flow runs in your pod, reads your warehouse over your network, writes your warehouse. Prefect learns that a task completed, not what was in it.
  • Your worker’s infrastructure credentials. The kubeconfig, the Docker daemon socket, the IAM role the pod assumes — those are the worker’s, held locally, never uploaded. (The exception you opt into: a push pool needs a scoped credential to call your cloud’s API on your behalf, which is exactly the trade you’re making when you choose “no worker.”)

What does cross — and be honest about this list:

  • Run metadata. Flow-run and task-run records, state names, timings, retry counts. Obviously.
  • Your flow parameters. They’re stored on the API so a run can be re-run and inspected. Corollary: do not pass secrets as flow parameters. People do this and then act surprised. Parameters are visible in the UI, forever.
  • Logs, if you let them (they’re shipped to the API by default). If your log lines contain customer PII, they are now in Prefect’s database. That’s a PREFECT_LOGGING_TO_API_ENABLED decision and a log-hygiene decision, and it’s yours.
  • Artifacts (chapter 17) — you published them precisely so someone could read them in the UI. That’s the point, but it means the table you rendered lives on the control plane.
  • Events you emit. Whatever you put in emit_event(payload=...) is stored.
  • Block documents. Here is the one people get wrong. A block is stored on the API — that’s what makes it loadable by name from anywhere. So when you write Secret(value="hunter2").save("warehouse-password"), that value goes over the wire and into Prefect’s database. Cloud encrypts it at rest and the UI obfuscates the field, and that is a perfectly reasonable place to keep it for most teams — but the sentence “Prefect never sees my credentials” stops being true the instant you put one in a block, and you should not say it to your security team.

The clean version of that last point, since it’s the one that matters: if your policy is that a third party must never hold a copy of a credential, then don’t put credentials in blocks. Use a block that holds a reference and resolves at run time inside your environment — an AwsSecret-style block that stores a secret name and fetches the value from your own Secrets Manager, using the worker’s own IAM role — or skip blocks entirely for that credential and inject it as an environment variable via your own infrastructure (chapter 07 is where the block types live). The block then contains a pointer, the pointer is not sensitive, and the boundary you described to your auditor is the boundary that actually exists.

Which sets up the comparison that ought to be doing the work in your evaluation. MWAA and Cloud Composer are not the same shape as Prefect Cloud. They run your scheduler and your workers — your DAG files are uploaded to them, your task code executes on their managed compute. Prefect Cloud runs the control plane only; the data plane stays entirely yours. For a regulated shop, that is a genuinely different conversation, and it’s the exact opposite of what people assume “SaaS orchestrator” means. The instinctive objection to Cloud — “we can’t send our data to a vendor” — is answered by the architecture, not by a promise. You aren’t sending your data anywhere. You’re sending the fact that a job finished.

API keys and service accounts

Now the specific self-inflicted wound, because I have watched this one land more than once.

Cloud has two kinds of API key. A user API key authenticates as a person and inherits that person’s role. A service account key authenticates as a robot — a first-class, non-human identity with its own role, that has no mailbox, no laptop, and no exit interview.

Here is the failure mode, in full. Your CI deploys with PREFECT_API_KEY, and that key was generated eighteen months ago by Sarah, because Sarah set the pipeline up and it was a Tuesday and she had a key already. Sarah takes a job elsewhere. IT deprovisions her in Okta on her last day — exactly what a well-run IT department should do. SSO revokes her Prefect access, and her API key dies with her identity.

At 06:00 the next Monday, prefect deploy --all fails with a 401 in a CI job nobody owns, and the deploy that was supposed to ship the quarter’s model refresh doesn’t. It takes two hours to find, and the finding is embarrassing, because the root cause is that your production deployment pipeline was authenticating as a person — and the correct security behaviour of your identity provider is what took it down. You didn’t get hacked. You got offboarded.

The fix costs about ninety seconds. Create a service account in the Cloud UI, give it a role scoped to what CI actually needs (create deployments — not “Admin”), generate its key, and put that in your secret store:

# .github/workflows/deploy.yml
- name: Deploy Prefect flows
  env:
    PREFECT_API_URL: https://api.prefect.cloud/api/accounts/${{ vars.PREFECT_ACCOUNT_ID }}/workspaces/${{ vars.PREFECT_WORKSPACE_ID }}
    PREFECT_API_KEY: ${{ secrets.PREFECT_CI_SERVICE_ACCOUNT_KEY }}
  run: |
    uv run prefect deploy --all

Note what’s not there: no prefect cloud login. In CI you don’t need it — environment variables beat profiles (chapter 15), so exporting PREFECT_API_URL and PREFECT_API_KEY is the whole authentication step. prefect cloud login -k <key> is a convenience for humans that writes those two values into a profile; a CI runner doesn’t want a profile, it wants a process environment.

Your workers need the same treatment, and for the same reason. A worker is a long-lived robot; it should hold a service account key with a worker-scoped role, not a human’s key with Admin on it. If a worker’s key leaks, you want the blast radius to be “can poll a work pool and report run states,” not “can delete every deployment in the workspace.”

Rotation is where the service-account model quietly earns its keep. Because the identity and the credential are separate objects, you can rotate without an outage:

  1. Generate a second key on the same service account.
  2. Roll it into your secret store; let the next deploy / worker restart pick it up.
  3. Confirm it’s in use.
  4. Delete the old key.

Two valid keys exist at once for a window, so nothing has to be simultaneous. Set an expiry on keys when you create them, so that a forgotten key eventually stops working on purpose rather than living forever as a credential nobody can account for.

Now hold that whole section up against the self-hosted server, where none of it exists. There is one password. It is in every worker’s environment, every CI secret, and every developer’s shell history. It cannot be scoped, it cannot be attributed, and rotating it is a synchronised restart of everything you own. If your answer to “how do we rotate credentials” is currently “we don’t,” self-hosting will not force you to fix that — it will just make it structurally impossible to.

The decision, then

Three questions. Answer them honestly and the choice usually makes itself.

1. Do you need per-user identity? Not “would it be nice.” Do you need to answer who cancelled that run, or to revoke one person’s access without changing everyone’s password, or to give a stakeholder read-only access that is actually read-only? If yes, the self-hosted server cannot get you there. There is no setting, no plugin, no prefect.toml table. Stop evaluating the two options; you have one.

2. Is the event trail forensically load-bearing? If a post-mortem three weeks after the fact is a normal part of your operating rhythm — because you’re in a regulated industry, or because your data is used for decisions people later audit — then a 7-day default on a disk you have to grow yourself is a liability with a fuse on it. You can fix it on self-hosted. You have to remember to, before you need it, which is a different thing from it being fixed.

3. Who is on call for the control plane at 3 a.m.? Say the name out loud. If the honest answer is “the data team,” then you have just made the data team a platform team, and you should price that: the Postgres, the ingress, the TLS certs, the upgrade path, the backups, the restore drill, the disk growth, and the pager. That is not zero, and it is not one-off — it recurs every quarter, forever, and it competes directly with the pipelines they were hired to build.

There’s a fourth question that isn’t really a question: does policy forbid a SaaS control plane? Data residency, an air-gapped network, a vendor review you will not win. If so, you’re self-hosting, the first three questions are moot, and everything in the self-hosting section above is your build list. Go do it well, and turn on PREFECT_SERVER_API_AUTH_STRING on your way past.

For most teams — a handful of data engineers, no compliance mandate, no air gap — the honest arithmetic is that Cloud’s lower tiers cost less than the engineering week you’d spend building the self-hosted equivalent, and the engineering week recurs. Self-host when you already have a platform team that runs Postgres and ingress and considers this a Tuesday, or when policy leaves you no choice. Those are good reasons. “It’s free” is not one, because it isn’t.

And the escape hatch is cheaper than the decision feels. Your flow code does not change between the two — this has been true since chapter 01 and it’s still true here; you’re pointing a client at a different URL. There’s even a supported move for the state:

prefect transfer --from self-hosted --to cloud-prod

Two profiles, one command, and your blocks, variables, deployments and work pools come across. Start self-hosted and graduate. Start on Cloud and pull it in-house when your platform team is ready. This is not a one-way door, and treating it like one is how evaluations turn into six-month stalls.

The Airflow trade, closed

The series has been an Airflow comparison from the first line, so let’s close this one properly, because the reflex answer is wrong in an interesting way.

Airflow’s open-source story is a whole platform you must run: a scheduler, an api-server, a metadata database, an executor, a fleet of workers, and a mechanism to get DAG files onto all of them. Running it seriously is a full-time job for at least one person, which is precisely why MWAA, Cloud Composer, and Astronomer exist and why “we self-host Airflow” so often means “two engineers whose actual job is Airflow.”

Prefect’s open-source story is a lighter control plane you must run: an API, a database, a UI, some loop services. You run workers either way, on both stacks and both editions, so they cancel out. Net: self-hosting Prefect is genuinely less work than self-hosting Airflow. That’s real, and it’s the argument you’ll hear.

Here’s the argument you won’t hear, and it’s the one that ought to change your mind. Self-hosted Airflow ships real users and real roles — its RBAC has Admin, Op, User, Viewer, Public, per-user accounts, and a permission model you can bend. Self-hosted Prefect ships one password. So if you are today running open-source Airflow with named accounts and a Viewer role for your analysts, and you migrate to a self-hosted Prefect server, you have just downgraded your access control — and I have never once seen that appear on the migration slide.

That’s the trade, stated without a thumb on the scale: Airflow makes you run more and gives you more governance for free. Prefect makes you run less and sells the governance back to you. Which of those is better depends entirely on whether you have a platform team and whether you have a compliance officer — and it is not obviously, universally, one of them.

Final thoughts

Go back to the slide with two columns on it. The correction isn’t to add a dollar figure to the left-hand side, though you should. It’s that the columns were never “free vs. paid” — they were “we operate the control plane” vs. “Prefect operates the control plane,” and the second column has always carried a price tag because the first one does too. Yours is simply paid in engineer-weekends, upgrade windows, and the disk you didn’t grow until the events were already deleted.

Choose either. Both are legitimate, and a competent team can run the self-hosted server well — Postgres, a migration job, an auth string, a proxy that forwards websockets, ninety days of event retention, and a restore drill somebody has actually performed. What you cannot do is choose self-hosted because it’s free and then act surprised that you bought a system instead of a licence. Price it honestly, name the person who gets paged, and the decision stops being a religious argument and becomes what it always was: a question about where your team’s scarce hours are best spent.

Next: Migrating from Airflow to Prefect

Comments