Work Pools and Workers

Prefect's scale-out story looks like Airflow's executors turned inside out: the API schedules runs into a pool, and independent workers pull them and provision infrastructure per run — unless the pool is serverless and there's no worker at all.

When serve() runs out of road — one host, one environment, one process that has to stay alive — the flow needs somewhere else to execute, and something else to decide where. That’s the job of two pieces you haven’t met yet: a work pool and a worker. Together they’re Prefect’s answer to the question Airflow answers with executors, and the shape of the answer is nearly the mirror image.

Pools and workers, defined

A work pool is a typed queue of scheduled runs. When a deployment’s schedule comes due, the server doesn’t run anything — it creates a flow run and drops it in the pool the deployment targets. The pool has a type — process, docker, kubernetes, ecs, and others — and that type dictates what kind of infrastructure runs get executed on. The pool itself is passive: a labeled inbox of pending work, plus a template describing how work from it should be run.

A worker is the process that empties it. You start one and point it at a pool:

prefect worker start --pool k8s-pool

From then on it polls that pool, and when a run appears it provisions the pool’s kind of infrastructure — a subprocess, a Docker container, a Kubernetes pod — runs the flow inside it, reports state back to the API, and tears the infrastructure down. Deployments target the pool by name; workers serve the pool by name; the two never reference each other directly. You can run several workers against one pool for capacity and redundancy, and one dead worker doesn’t strand the pool — another picks up the runs.

The Airflow contrast: pull, not push

If you’ve read the Airflow executors post, the difference is worth drawing sharply, because it’s structural rather than cosmetic.

Airflow’s executor is wired into the scheduler. The scheduler decides a task is ready and the executor — Local, Celery, Kubernetes — dispatches it: Local runs it in a subprocess of the scheduler, Celery pushes it to a broker for workers to consume, Kubernetes asks the API for a pod. The executor is a property of the scheduler, chosen once for the whole deployment, and the scheduler is upstream of every task placement.

Prefect cuts that cord. The API’s only job is to put a scheduled run in a pool; it does not dispatch, does not know or care whether anything is listening. Workers are independent, long-running clients that pull from a pool on their own clock. Nothing centrally decides that this worker runs that flow — a worker takes the next run when it’s ready for it. Scheduling and execution end up fully decoupled: the API can file runs into a pool with no workers up at all, and they’ll wait until a worker starts and drains them. Airflow pushes work down from the scheduler through its one executor; Prefect parks work in a pool and lets workers pull it, and you can have different pools, of different types, served by different workers, all at once.

The practical upshot is per-run infrastructure that isn’t the scheduler’s concern. A Kubernetes work pool doesn’t mean “Prefect runs on Kubernetes” the way the KubernetesExecutor ties Airflow’s scheduler to a cluster — it means runs from that pool get a pod each, while a process pool on some other box handles the cheap stuff, and neither knows about the other.

Two families: pull pools and push pools

Everything above describes a pull work pool — the kind that needs a worker. There’s a second family that doesn’t, and skipping it leaves you thinking you always have a worker process to babysit. You don’t.

A push work pool hands submission to a cloud’s serverless container service. Instead of a worker polling and launching infrastructure, the Prefect API (Cloud only) calls the cloud provider’s API directly the moment a run is scheduled, and the provider spins up the container. Four flavours ship today:

  • ecs:push — AWS Fargate tasks, no ECS cluster capacity to manage.
  • cloud-run:push — Google Cloud Run jobs.
  • azure-container-instance:push — Azure Container Instances.
  • modal:push — Modal’s serverless runtime.

Create one exactly like a pull pool, only the type carries a :push suffix:

prefect work-pool create serverless-pool --type ecs:push --provision-infra

--provision-infra is the part that earns its keep: for a push pool, Prefect walks your cloud account and creates (or reuses) everything the pool needs — for ECS that’s a cluster, an IAM execution role and policy, a VPC/security group, and an ECR repository — then stores the credentials as a block and wires them into the pool. You answer a few “create this? [y/n]” prompts and the pool comes out ready to run, no Terraform of your own.

There is also Prefect-managed execution — prefect work-pool create my-pool --type prefect:managed — where Prefect Cloud runs the containers on its infrastructure. No cloud account, no worker, no --provision-infra; you deploy and runs happen. It’s the fastest path to a scheduled flow that isn’t tied to your laptop, with the tradeoff that you’re on Prefect’s compute (capped resources, no access to your private VPC without extra wiring).

The way to hold the three apart:

  • Pull pool — you run a worker; the worker launches infra in your environment. Maximum control, a process to keep alive.
  • Push pool — no worker; Prefect Cloud calls your cloud’s serverless API. Infra is yours, submission is Prefect’s.
  • Managed pool — no worker, no cloud account; runs execute on Prefect’s compute.

Cost and cold-start fall out of that. A pull pool with an idle worker is a machine you pay for even when nothing runs, but the worker is warm and submission is instant. A push pool costs nothing between runs — Fargate/Cloud Run bill per execution — at the price of a cold start each time the provider builds a container (seconds to tens of seconds, image-size dependent). For a flow that fires hourly, the always-on worker is cheap insurance; for one that fires twice a day, or bursts unpredictably, serverless push is usually the better economics. Isolation is strong in all three — every run is a fresh container regardless.

The base job template: where infrastructure is actually configured

Both families run on the same underlying mechanism, and it’s the piece people miss: every pool carries a base job template. It’s a JSON document with two halves — a job_configuration block (the infrastructure spec: image, command, env, CPU, memory, namespace, the manifest for a k8s pool) written with {{ placeholder }} fields, and a variables block (a JSON Schema) declaring which of those placeholders are tunable and their defaults. When a run is submitted, Prefect renders the template: variables fill the placeholders, and the result is the concrete infrastructure spec.

You can read the default for any type before creating a pool:

prefect work-pool get-default-base-job-template --type kubernetes

For a Kubernetes pool that prints the full Job manifest with {{ image }}, {{ namespace }}, {{ cpu_request }}, {{ service_account_name }} and friends threaded through it, plus the schema of defaults. Edit that JSON and pass it at create time, and you’ve set the pool’s baseline for every deployment on it:

prefect work-pool create k8s-pool \
  --type kubernetes \
  --base-job-template ./k8s-template.json

This is the answer to “how do I set the image / memory / service account for my runs” — you don’t hand-write pod YAML per flow, you set pool defaults in the template and override the handful that vary per deployment. Same idea as Airflow’s pod_template_file for the KubernetesExecutor, except it’s per-pool, versioned as data on the server, and editable in the UI.

Overriding per deployment and per run: job_variables

The template gives defaults; job_variables overrides them for a specific deployment. Anything the pool’s variables schema exposes — image, env, resource requests, namespace — you can set at deploy time:

from flows import daily_load

if __name__ == "__main__":
    daily_load.deploy(
        name="bookshop-daily",
        work_pool_name="k8s-pool",
        image="registry.example.com/bookshop:latest",
        job_variables={
            "namespace": "data-prod",
            "service_account_name": "prefect-runner",
            "env": {"DBT_TARGET": "prod"},
            "cpu_request": "1000m",
            "memory_request": "2Gi",
        },
    )

In prefect.yaml the same thing lives under the deployment’s work_pool:

deployments:
  - name: bookshop-daily
    entrypoint: flows/daily_load.py:daily_load
    work_pool:
      name: k8s-pool
      job_variables:
        namespace: data-prod
        memory_request: "2Gi"
        env:
          DBT_TARGET: prod

And you can go one level finer — override at the moment a single run is created, without touching the deployment. From the UI’s “Custom run” form, or from code:

from prefect.deployments import run_deployment

run_deployment(
    name="daily-load/bookshop-daily",
    job_variables={"memory_request": "8Gi"},  # this one run needs more headroom
)

So there are three layers, each overriding the last: the pool’s base job template, the deployment’s job_variables, and a single run’s job_variables. That layering is how you keep one pool serving many flows — shared defaults in the template, the differences declared where they belong.

Per-type operational detail

The three pull types differ in exactly the ways their infrastructure differs.

process runs each flow in a subprocess on the worker’s own host. There’s no image; the flow uses the worker’s Python environment, so the worker has to be started in a venv that already has your dependencies (and your flow’s source, unless the deployment pulls it from git/remote storage). job_variables here are things like env and working_dir. No isolation beyond the process, nothing to build, fast to start — the natural step up from serve().

docker runs each flow in a fresh container from an image you specify. The worker talks to a Docker daemon (its own host’s, typically), so it needs registry auth to pull a private image — you provide that with a DockerRegistryCredentials block referenced from the template, or by logging the daemon in out of band. prefect deploy can build and push the image for you as part of deployment (driven by the build/push steps in prefect.yaml), so the image the pool pulls is the one your code just produced. Every run gets its own dependencies and a clean filesystem; two flows with conflicting requirements coexist.

kubernetes runs each flow as a Job — a pod per run — on a cluster. The operational surface is the cluster’s: the pool template sets the namespace, the service account the pod runs as (and therefore its RBAC — the worker itself also needs a Role that can create/watch/delete Jobs and read pod logs), image pull secrets, and per-run CPU/memory requests and limits. This is the elastic, resource-governed option, and the most operational overhead — you’re now managing RBAC and namespaces, not just a template.

Deciding between the lot comes down to three axes. Isolation: process gives you none beyond a subprocess, everything else gives a fresh container per run — so anything untrusted, or with dependencies that clash, rules process out immediately. Cost: an idle pull-pool worker (and its host, or its k8s Deployment) is a standing bill whether or not flows run; push and managed pools bill only per execution. Cold-start: the warm worker submits instantly, while push/serverless pays a container-build penalty at the front of every run — negligible for a long ETL job, painful for a flow you want to feel interactive. A blunt heuristic: process for trusted, high-frequency, low-isolation work on a box you already keep alive; docker/kubernetes pull pools when you need isolation and run often enough that a warm worker pays for itself; ecs:push/cloud-run:push when runs are spiky or infrequent and per-run billing beats an always-on worker; prefect:managed when you want none of the above and Prefect’s compute will do.

Worker operations

Starting a worker is one command, but the flags are where day-two lives:

prefect worker start \
  --pool k8s-pool \
  --name worker-a \
  --type kubernetes \
  --limit 5 \
  --work-queue high
  • --pool is the only required flag. --type is optional — the worker infers it from the pool — but stating it makes the worker refuse a mismatched pool, which catches wiring mistakes.
  • --name gives the worker a stable identity in the UI; without it you get a generated one, which is fine until you’re staring at ten anonymous workers wondering which host each is on.
  • --limit N caps concurrent flow runs this worker will submit. On a process pool that’s how many subprocesses one host will run at once; it’s the per-worker backpressure valve.
  • --work-queue narrows the worker to one queue in the pool (more on queues below). Omit it and the worker serves every queue.

A running worker does three things on a loop. It polls the pool on an interval (PREFECT_WORKER_QUERY_SECONDS, default 10) for runs due to start. It prefetches — PREFECT_WORKER_PREFETCH_SECONDS (default 10) tells it to submit runs a little before their scheduled time, so slow-to-start infrastructure (a cold pod, an image pull) is ready by the wall-clock moment; bump it for a k8s pool where pods take a while to schedule. And it heartbeats — the worker regularly reports it’s alive, so the UI shows it online and the server can tell a worker apart from a dead one. Stop heartbeating and the worker goes offline in the UI; its runs don’t vanish, they wait in the pool for another worker.

Shutdown is graceful by design. Send SIGINT/SIGTERM (Ctrl-C, or a container stop) and the worker stops polling for new work but waits for runs it has already submitted, so you can redeploy a worker without killing in-flight flows — the same rolling-restart property you’d want from any long-lived poller.

For a --provision-infra push pool there’s no worker to run at all, but for pull pools in a cluster the worker is itself a workload you deploy. Prefect ships a Helm chart for exactly this:

helm repo add prefect https://prefecthq.github.io/prefect-helm
helm install prefect-worker prefect/prefect-worker \
  --namespace prefect --create-namespace \
  --set worker.cloudApiConfig.accountId=<account> \
  --set worker.cloudApiConfig.workspaceId=<workspace> \
  --set worker.config.workPool=k8s-pool

That runs the worker as a Deployment in the cluster, with the API key supplied as a secret — so the thing that launches your Job pods lives in the same cluster it launches them into, and Kubernetes keeps it alive and restarts it for you.

The prefect work-pool CLI

The pool is a first-class object with its own verbs. The ones you’ll actually use:

prefect work-pool create k8s-pool --type kubernetes   # make one
prefect work-pool ls                                   # list pools + worker status
prefect work-pool inspect k8s-pool                     # full config incl. base job template
prefect work-pool preview k8s-pool                     # runs this pool will pick up soon
prefect work-pool set-concurrency-limit k8s-pool 10    # cap concurrent runs across the pool
prefect work-pool pause k8s-pool                       # stop handing out runs (they queue)
prefect work-pool resume k8s-pool

set-concurrency-limit is the pool-wide governor — a ceiling on how many runs from this pool execute at once, no matter how many workers are pulling. It’s the knob that keeps a pool from overwhelming a downstream (a warehouse that tolerates ten concurrent loads, not fifty). pause is the quiet-hours switch: a paused pool keeps accepting scheduled runs but stops dispatching them, so runs pile up safely and drain the moment you resume — the clean way to hold execution during a maintenance window without losing a schedule.

Inside a pool, work queues give finer control still. A pool has a default queue, and you can add named queues with priorities (a lower number is higher priority — a high queue at priority 1 is drained before the default) and their own concurrency limits:

prefect work-queue create high --pool k8s-pool --priority 1 --limit 3

Now a deployment can target high, a worker can be pinned to it with --work-queue high, and workers drain higher-priority queues first — so a greedy backfill on the default queue can’t starve the morning load. Reach for queues when one pool serves work of genuinely different urgency; skip them until you feel that need.

The hybrid security model

One property makes all of this deployable behind a firewall, and it’s worth stating outright because it’s the opposite of how a naive “cloud runs my jobs” system would work. Workers only make outbound calls. A worker polls the Prefect API over HTTPS, pulls down the runs it should execute, and reports states back — it never listens on a port, and nothing (not even Prefect Cloud) connects in to it. You can run workers deep inside a private VPC with no ingress and they still work, because they reach out.

The corollary is that your code and credentials never leave your environment on a pull pool. Prefect Cloud holds metadata — deployments, schedules, run states, logs you choose to send — but the worker executes in your infrastructure, pulls your image from your registry, and uses secrets from your own blocks/secret store. Cloud never holds your database password or your cloud keys. (Push pools shift that line deliberately: to call your cloud’s API on your behalf, Prefect stores a scoped credential from --provision-infra — a tradeoff you opt into for the no-worker convenience.) This is the same posture Airflow deployments reach for the hard way with a private executor and a locked-down network; in Prefect it’s the default shape, because the worker is a client, not a server.

The bookshop on Kubernetes

Concretely, the daily bookshop load on a Kubernetes pool. Create the pool once, with a template that sets the sensible defaults:

prefect work-pool create k8s-pool \
  --type kubernetes \
  --base-job-template ./k8s-template.json

Deploy the flow to it, overriding only what’s specific to this flow:

from flows import daily_load

if __name__ == "__main__":
    daily_load.deploy(
        name="bookshop-daily",
        work_pool_name="k8s-pool",
        image="registry.example.com/bookshop:latest",
        job_variables={"memory_request": "2Gi", "env": {"DBT_TARGET": "prod"}},
    )

Run a worker in the cluster (or install the Helm chart), pointed at the pool:

prefect worker start --pool k8s-pool --name k8s-worker-a

Now the pieces move on their own. At 6 a.m. the server creates a run and files it in k8s-pool. The worker, polling — and prefetching a few seconds early so the pod is ready on time — sees it, renders the base job template with your job_variables, asks Kubernetes for a pod running bookshop:latest, and the pod runs daily_load, including the concurrent .map() fan-out from the dynamic execution post, inside that one pod. It reports its states back and exits. Tomorrow, a fresh pod. Nothing of yours stayed up overnight but the worker, and even that can be replaced without losing a scheduled run, because the run was safe in the pool the whole time.

Want the same flow with no worker at all? Point the deployment at an ecs:push pool instead and delete the prefect worker start line — Prefect Cloud submits the Fargate task the instant the run is scheduled, you pay per run, and the only thing you gave up is the warm poller.

Final thoughts

Work pools and workers are the same responsibility as an Airflow executor — get runs onto infrastructure — split along a different seam. Airflow keeps execution close to the scheduler, one executor for the whole deployment, work pushed down from the top. Prefect files runs into a typed pool and lets the edge decide how they run: a worker pulls them and stands up disposable infrastructure per run, or, on a push pool, no worker exists and Prefect Cloud calls your cloud’s serverless API directly. The base job template is where the infrastructure actually lives, job_variables is how you bend it per deployment and per run, and the hybrid model — workers polling outbound, credentials never leaving your environment — is what lets all of it sit behind a firewall. It’s more moving parts than a single executor setting, and in exchange the thing that decides when a flow runs stops being coupled to the thing that decides where — and you get to choose, pool by pool, whether where is even a machine you keep running.

Next: Blocks: Config and Secrets

Comments