Automations: The 3 a.m. Runbook That Runs Itself

Here’s a question worth sitting with. What does Airflow do when a DAG fails?

The honest answer is: nothing. Airflow marks the run red and moves on. Everything else — the Slack message, the PagerDuty page, the decision to stop scheduling more runs of a pipeline that’s clearly broken, the recovery job, the “don’t re-run this until someone looks at it” — is something you built, inside the DAG, out of on_failure_callback functions and cleanup tasks and trigger rules. Your reaction to failure is a subgraph. It lives in the same file as the work, it deploys with the work, it fails with the work, and if the pod that was running the DAG got OOM-killed, your on_failure_callback — which is Python, in that pod — got killed with it. The alerting you built to tell you about infrastructure failures does not survive infrastructure failures.

Prefect’s answer is a different shape entirely. Prefect emits an event for essentially everything that happens — every state change, every deployment, every work pool going quiet — into a stream the server owns. And an automation is a rule over that stream: a trigger that watches for a pattern and a list of actions that fire when it matches. The rule lives on the server. It is not in your flow, it does not deploy with your flow, and it does not die when your flow’s process does. It is a control plane that sits outside your pipelines and watches them.

That relocation — reactions out of the DAG, onto the platform — is the whole chapter.

Everything is an event

Start with the substrate, because the trigger grammar only makes sense once you’ve seen what it’s matching against.

A Prefect event is a small, uniform record with a fixed shape:

{
  "occurred": "2026-06-08T02:14:03.221Z",
  "event": "prefect.flow-run.Failed",
  "resource": {
    "prefect.resource.id": "prefect.flow-run.9f2c…",
    "prefect.resource.name": "bookshop-extract",
    "prefect.state-name": "Failed"
  },
  "related": [
    {"prefect.resource.id": "prefect.deployment.4a1b…", "prefect.resource.role": "deployment"},
    {"prefect.resource.id": "prefect.work-pool.7cd2…", "prefect.resource.role": "work-pool"}
  ],
  "payload": {},
  "id": "…"
}

Five fields do all the work. event is the name — a dotted string, and for Prefect’s own events it’s almost always prefect.<thing>.<StateName>. resource is the primary thing the event is about, identified by prefect.resource.id, with whatever labels are relevant hanging off it. related is everything else implicated: the deployment the run belongs to, the work pool it ran in, the flow it’s an instance of, each tagged with a role. payload is free-form detail. And occurred is when.

Every state transition from the states chapter has a corresponding event: prefect.flow-run.Running, prefect.flow-run.Completed, prefect.flow-run.Failed, prefect.flow-run.Crashed, prefect.flow-run.Cancelled, and the same family for task-run. There are operational ones too — deployments, work pools, work queues going healthy and unhealthy. Plus whatever you emit yourself, which we’ll get to.

You can watch the whole thing go by, live, which is the fastest way to learn the event names you’ll be writing triggers against:

prefect events stream                     # JSON, live, every event
prefect events stream --format text

Run that in one terminal, kick a flow in another, and you’ll see the run’s whole life scroll past as events. Do this once before you write your first automation. Every trigger you write is a pattern-match against those exact strings, and guessing at them from memory is how you end up with an automation that has never fired and that you’ve assumed for six months is protecting you.

An automation is a trigger plus actions

The object model is small enough to state in a sentence: an automation has a name, one trigger, and a list of actions. When the trigger fires, the actions run, in order.

The trigger is one of exactly four types — event, metric, compound, sequence — and the actions come from a fixed catalog of eighteen. That’s the whole vocabulary. What makes it expressive isn’t the size of the catalog; it’s that the trigger grammar composes and the actions compose, so a handful of primitives covers an enormous amount of ground.

Event triggers, and the grammar you’ll use daily

The event trigger is the one you’ll write ninety percent of the time. Its fields:

expect — the set of event names that arm it. {"prefect.flow-run.Failed"}.
match — a filter on the event’s own resource. {"prefect.resource.name": "bookshop-extract"}.
match_related — a filter on the related resources. This is how you scope to a deployment or a work pool, because a flow-run event’s own resource is the run, and the deployment is related to it. Getting match and match_related backwards is the single most common reason a trigger never fires.
after — an event that must have been seen first to arm the trigger. after={"prefect.flow-run.Running"} with expect={"prefect.flow-run.Failed"} means “a failure that followed a start.”
for_each — the resource label to group by. This is the underrated one. for_each={"prefect.resource.id"} says: count thresholds per run, not across all runs. Without it, “three failures” means three failures of anything matching; with it, it means three failures of the same resource.
threshold — how many matching events are required. Default 1.
within — the window the threshold is measured over, as a timedelta. Default zero.
posture — Reactive or Proactive. This is the big one.

Reactive means: fire when the expected event arrives. That’s the normal case — something bad happened, do something.

Proactive means: fire when the expected event does not arrive within within. This is a dead man’s switch, and it is the trigger Airflow engineers most conspicuously lack. Airflow removed SLAs in 3.x and never replaced the “this thing was supposed to happen and didn’t” primitive. Prefect just inverts the posture:

- name: nightly-load-never-finished
  trigger:
    type: event
    posture: Proactive
    after: ["prefect.flow-run.Running"]
    expect: ["prefect.flow-run.Completed"]
    match_related:
      prefect.resource.id: "prefect.deployment.4a1b…"
    within: 5400        # 90 minutes
  actions:
    - type: send-notification
      block_document_id: "{{ oncall_slack }}"
      subject: "bookshop nightly load has not completed"
      body: "It started, and 90 minutes later it still hasn't finished or failed."

Read the semantics carefully, because they’re precise and they’re good: the trigger arms when the run starts (after), then waits for Completed (expect), and fires only if ninety minutes pass without it. A run that finishes in forty minutes fires nothing. A run that fails fires nothing (that’s a different automation’s job). A run that hangs — the case that silently ruins your morning, because a hung run is neither red nor done — fires this. That’s the alarm you could never quite build in Airflow without a second DAG whose whole job was to go and check on the first one.

The reactive counterpart, with a threshold, in Python this time:

from datetime import timedelta
from prefect.automations import Automation
from prefect.events.schemas.automations import EventTrigger, Posture
from prefect.events.actions import RunDeployment

Automation(
    name="recover-after-three-failures",
    trigger=EventTrigger(
        expect={"prefect.flow-run.Failed"},
        match_related={"prefect.resource.id": "prefect.deployment.4a1b…"},
        posture=Posture.Reactive,
        threshold=3,
        within=timedelta(minutes=10),
    ),
    actions=[RunDeployment(deployment_id=recovery_id)],
).create()

Three failures of that deployment inside ten minutes, and the recovery deployment starts. One failure does nothing — because one failure is a Tuesday, and three in ten minutes is an incident.

`match` versus `match_related`, which you will get wrong once

Spend thirty seconds on this and save yourself an afternoon. A prefect.flow-run.Failed event’s own resource is the flow run — its id, its name, its state. The deployment it came from, the flow it’s an instance of, and the work pool it ran in are all in related, each stamped with a prefect.resource.role.

So this trigger never fires:

# WRONG — a flow-run event's resource is the RUN, not the deployment
match:
  prefect.resource.id: "prefect.deployment.4a1b…"

and this one does:

# RIGHT — the deployment is a RELATED resource of the event
match_related:
  prefect.resource.id: "prefect.deployment.4a1b…"

The rule of thumb: match filters the noun the event is about; match_related filters everything that noun belongs to. Scoping to “runs of this deployment,” “runs in this work pool,” or “runs of this flow” is always match_related. Scoping by the run’s own name or state is match. Both accept lists of values, so match_related: {prefect.resource.name: ["bookshop-extract", "bookshop-transform"]} covers two deployments in one rule.

Test the trigger without waiting for a real failure

An automation you’ve never seen fire is an automation you don’t have. You can hand-inject an event and watch the thing go off:

prefect events emit bookshop.orders.loaded \
  --resource-id "bookshop.orders.2026-06-08" \
  --payload '{"day": "2026-06-08", "rows": 41233}'

That’s a real event in the real stream, and any automation matching it fires for real. Pair it with prefect events stream in another terminal and you have a complete loop: emit, watch it land, watch the action run. Do this for every automation you write, especially the destructive ones — because the alternative is discovering that your cancel-flow-run rule matches more broadly than you thought, at the exact moment it cancels something you needed.

Metric triggers: when counting events is the wrong instrument

Some questions are not about events at all. “Has our success rate dropped below 90% this hour?” is not a pattern you can express by counting Failed events, because the answer depends on how many runs there were. “Is the p95 duration double what it was?” isn’t an event either. That’s what the metric trigger is for, and it watches exactly three metrics — no more:

successes — the success rate, as a fraction.
duration — how long runs are taking.
lateness — how far behind schedule runs are starting.

The query has four parts: the metric name, an operator (<, <=, >, >=), a threshold, a range (the window the metric is computed over, default 5 minutes), and firing_for (how long it must stay bad before the automation fires, also default 5 minutes).

- name: bookshop-success-rate-dropped
  trigger:
    type: metric
    match_related:
      prefect.resource.id: "prefect.deployment.4a1b…"
    metric:
      name: successes
      operator: "<"
      threshold: 0.9        # below 90% success…
      range: 3600           # …measured over the trailing hour…
      firing_for: 600       # …and still bad ten minutes later
  actions:
    - type: send-notification
      block_document_id: "{{ oncall_slack }}"

The range / firing_for pair is what separates a useful metric trigger from a pager that everyone mutes by Thursday. range is the lens; firing_for is the patience. Set firing_for: 0 and a single unlucky minute wakes someone up. Set it to ten minutes and you’re saying “I’ll believe it when it’s still true after ten minutes,” which is what a human on-call would do anyway, so encode it.

Compound and sequence triggers: correlation and order

The last two trigger types don’t watch events directly — they watch other triggers, which lets you express conditions that no single event can carry.

A compound trigger has a list of child triggers, a within window, and a require of "any", "all", or an integer n: fire when at least that many children have each fired inside the window.

- name: bookshop-extract-incident
  trigger:
    type: compound
    require: all           # or "any", or 2
    within: 600
    triggers:
      - type: event
        expect: ["prefect.flow-run.Failed"]
        match_related:
          prefect.resource.name: "bookshop-extract"
      - type: event
        expect: ["prefect.work-queue.unhealthy"]
  actions:
    - type: run-deployment
      deployment_id: "{{ recovery_deployment_id }}"

The extract failing is noise; you have retries for that. The work queue going unhealthy on its own is noise too; it happens during deploys. Both, within ten minutes is an incident, and require: all is how you say so. This is correlation as a first-class trigger, and it’s the difference between an alerting setup people trust and one they’ve filtered into a folder.

A sequence trigger is a compound trigger that also demands order: child A must fire, then B, then C, all within the window. And the distinction is not pedantry — it’s the difference between two opposite realities built from the same two events:

- name: run-is-stuck-not-recovering
  trigger:
    type: sequence
    within: 1800
    triggers:
      - type: event
        expect: ["prefect.flow-run.Running"]
      - type: event
        expect: ["prefect.flow-run.Late"]
  actions:
    - type: cancel-flow-run

A run that went Late and then Running is a run that was queued too long and is now recovering — leave it alone. A run that went Running and then Late is a run that started, lost its worker, and is now a zombie holding a slot. Same two events. Opposite meanings. Only sequence can tell them apart, and a compound with require: all would fire on both, which is how you end up with an automation that cancels healthy runs.

The actions catalog

When a trigger fires, its actions list executes. The full catalog, which is worth reading in one go because half of these are things people don’t realize they can do without writing code:

Action	What it does
`run-deployment`	Start a run of a deployment, with `parameters` and `job_variables`
`pause-deployment` / `resume-deployment`	Stop or restart a deployment’s schedule
`pause-work-queue` / `resume-work-queue`	Quarantine or restore a queue
`pause-work-pool` / `resume-work-pool`	Quarantine or restore a whole pool
`cancel-flow-run`	Kill a run
`suspend-flow-run`	The softer stop — frees infrastructure, resumable (chapter 16)
`resume-flow-run`	Resume a paused/suspended run — an automation can answer a gate
`delete-flow-run`	Remove a run outright
`change-flow-run-state`	Force a run into a state, with a `message` and a `force` flag
`send-notification`	Send via a notification block
`call-webhook`	POST to an external system via a webhook block
`pause-automation` / `resume-automation`	Automations arming and disarming each other
`declare-incident`	Open a tracked incident (Cloud only)
`do-nothing`	Exactly what it says — useful as a placeholder while you build

Two of these deserve a second look. resume-flow-run means an automation can supply the answer to a human-in-the-loop gate — a Suspended run can be resumed by a rule, not just a person, which turns “wait for approval” into “wait for approval or for this other condition to become true.” And pause-automation means automations can regulate each other: an incident automation that disables the noisy per-failure alerter while the incident is open is four lines of YAML, not a feature request.

Note also that the automation object has actions, actions_on_trigger, and actions_on_resolve. That last one is the “all clear” hook — for a proactive or metric trigger that stops firing, actions_on_resolve lets you send the “recovered” message that closes the loop. An alert that never tells you it’s over is an alert you’ll eventually stop reading.

Composing actions into a runbook

The catalog is fine. The composition is the point. Actions run in order, so one trigger can execute a sequence that a human would otherwise be doing by hand, half-awake, under pressure:

- name: bookshop-extract-runbook
  trigger:
    type: event
    expect: ["prefect.flow-run.Failed"]
    match_related:
      prefect.resource.name: "bookshop-extract"
    for_each: ["prefect.resource.id"]
    threshold: 3
    within: 600
  actions:
    # 1. stop digging — no more scheduled runs while this is broken
    - type: pause-deployment
      deployment_id: "{{ extract_deployment_id }}"

    # 2. kick the recovery path, and tell it WHICH run died
    - type: run-deployment
      deployment_id: "{{ recovery_deployment_id }}"
      parameters:
        failed_run_id: "{{ event.resource['prefect.resource.id'] }}"

    # 3. tell a human what already happened without them
    - type: send-notification
      block_document_id: "{{ oncall_slack }}"
      subject: "bookshop-extract paused after 3 failures"
      body: >
        Deployment paused. Recovery started.
        Failed run: {{ event.resource['prefect.resource.id'] }}

Read those three actions as three lines of an incident checklist. Stop the bleeding. Start the recovery. Tell someone. That’s a runbook, and it executes itself in the second after the third failure rather than the twenty minutes after somebody notices a red square.

The {{ event.… }} templating is what elevates this from “notifier” to “responder.” Actions render Jinja against the triggering event, so the recovery deployment is told which run died instead of having to go and figure it out, and the Slack message contains the actual run id instead of “something failed, go look.” An automation that only notifies is a smoke alarm. An automation that templates the event into its actions is a sprinkler.

Notification blocks

The send-notification action doesn’t take a Slack URL — it takes a block_document_id. Which means the credential lives exactly once, in a block, and every automation references it. Rotate the webhook, and nothing else changes.

from prefect_slack import SlackWebhook

SlackWebhook(url="https://hooks.slack.com/services/…").save("oncall", overwrite=True)

from prefect_pagerduty import PagerDutyWebHook

PagerDutyWebHook(
    integration_key="…",
    api_key="…",
).save("oncall-page", overwrite=True)

Slack, PagerDuty, Microsoft Teams, email via EmailServerCredentials, Twilio, Opsgenie — the notification blocks come from the integration libraries and all of them plug into the same action. The subject and body are Jinja-templated against the event, so you can route different severities to different blocks and give each one a message that says something useful.

The rule I’d hold to: Failed goes to the team that owns the pipeline; Crashed goes to the team that owns the infrastructure. Prefect draws that distinction in the state model (the whole point of chapter 13), and automations are where you finally use it — two automations, two expects, two blocks. Airflow collapses both into failed and routes them to the same overloaded channel, where they get equally ignored.

`emit_event()`: your own events in the same stream

Everything so far reacts to Prefect’s events. But the stream isn’t Prefect’s — it’s yours, and you can push domain events into it from any code that can reach the API:

from prefect import task
from prefect.events import emit_event

@task
def load(staged: str, day: str) -> int:
    rows = load_warehouse(staged)
    emit_event(
        event="bookshop.orders.loaded",
        resource={
            "prefect.resource.id": f"bookshop.orders.{day}",
            "prefect.resource.name": "bookshop orders",
            "bookshop.day": day,
        },
        payload={"rows": rows, "day": day},
    )
    return rows

bookshop.orders.loaded is now a first-class event that any automation can expect, match, and template against, exactly like a built-in one. The resource dict needs a prefect.resource.id and can carry any other labels you like; the payload carries the detail.

This is the move that changes what automations are for. Without it, you’re building an ops layer: alert on failure, restart on crash. With it, you’re building an event-driven data platform: “when the orders for a day load, refresh the dashboards for that day,” expressed as a rule between two independently-deployed pipelines that don’t import each other, don’t know each other’s names, and can be changed independently.

- name: refresh-dashboards-when-orders-land
  trigger:
    type: event
    expect: ["bookshop.orders.loaded"]
  actions:
    - type: run-deployment
      deployment_id: "{{ dashboard_refresh_id }}"
      parameters:
        day: "{{ event.payload['day'] }}"

If you’ve read the Airflow assets chapter, this is the same instinct — schedule on data, not on the clock — with the ceiling removed. Airflow’s asset scheduling is one event type (“a dataset was produced”) wired to one action (“run a DAG”). Prefect’s is any event you can name, wired to any of eighteen actions, and the coupling between the two pipelines is a rule on the server rather than an object both DAGs must import. That last part matters more than it sounds: it means the downstream team can change what happens when orders land without opening the upstream repo.

Inbound webhooks: events from systems that aren’t yours

The other direction. Your vendor drops a file. Your CI finishes a build. Your SaaS tool fires a callback. None of these can call emit_event() — they’ve never heard of Prefect, and they’re not going to.

An inbound webhook is a static HTTPS endpoint Prefect hosts for you. The third party POSTs to it. A Jinja template you configure turns the request body into a Prefect event, which lands in the stream and arms your triggers like anything else:

prefect cloud webhook create vendor-drop \
  --template '{
    "event": "vendor.file.dropped",
    "resource": {
      "prefect.resource.id": "vendor.drop.{{ body.dataset }}",
      "vendor.path": "{{ body.s3_path }}"
    }
  }'

You get back a URL, you hand it to the vendor, and the next time they finish their nightly drop your ingest deployment starts within a second — no polling sensor, no cron guessing when the file lands, no FileSensor in reschedule mode occupying a slot for six hours because the vendor was late.

Two things to be straight about. First, the CLI is prefect cloud webhook, and that prefix is not an accident: inbound webhooks are a Prefect Cloud feature. They need a stable, externally-addressable, authenticated URL, and OSS Server doesn’t provide one. Second, this is the first place in this chapter where the Cloud/OSS line actually bites, so let’s draw it properly.

Cloud versus OSS Server, honestly

This changed between Prefect 2 and Prefect 3, and a lot of writing on the internet is still describing the old world, so here is the current state:

Events and automations run on the self-hosted open-source Server. Not Cloud-only. Run prefect server start and you get the event stream, most of the trigger taxonomy, and the action catalog. In the 2.x era this was largely a Cloud story; in 3.x it isn’t. If you’re evaluating Prefect and someone tells you automations are a paid feature, they are describing 2022.

What stays Cloud-only: metric triggers — the OSS server has no MetricTrigger at all, and posting one to /api/automations/ comes back 422 Unprocessable Entity, which is a confusing way to learn this. (Event, compound, and sequence triggers all create fine.) Also Cloud-only: inbound webhooks (the hosted URLs above), the declare-incident action and the incident-management surface around it, and the long-term event retention and SLA guarantees.

One retention detail worth setting a calendar reminder about, because it is the kind of thing you discover during a post-mortem: the OSS server keeps events for 7 days by default (PREFECT_SERVER_EVENTS_RETENTION_PERIOD). Automations fire off the event stream, and your forensic trail is the event stream — so if you plan to ask “what happened three weeks ago,” raise that number before you need it, not after. Plus the usual managed-platform things — SSO, RBAC, audit logs, managed work pools.

So the practical shape: you can develop, test, and run automations entirely on OSS. You reach for Cloud when you need external systems to push events in, when you want managed incidents rather than rolling your own, or when you need event history to persist longer than a self-hosted database you have to babysit. That’s a real and defensible line, and it’s a much smaller Cloud tax than Prefect 2 charged.

Automations as code

Clicking an automation together in the UI is a fine way to learn and a terrible way to operate. An automation is infrastructure: it pauses your deployments, it cancels your runs, it pages your people. It belongs in the repo, in review, in the deploy.

Three ways in, all equivalent:

A YAML file plus the CLI, which is my default because it diffs cleanly and a reviewer can read it:

prefect automation create --from-file automations.yaml
prefect automation ls
prefect automation inspect bookshop-extract-runbook
prefect automation pause bookshop-extract-runbook     # disable without deleting

Python, when the automation needs to be built from values you have to look up anyway (a deployment id, a block document id) — Automation(...).create(), as shown earlier. It has the full complement: .create(), .read(), .update(), .delete(), .disable(), .enable(), and a-prefixed async twins for all of them.

Inline in prefect.yaml, under a deployment’s triggers: key, which is the tightest coupling and often the right one — the rule that protects a deployment ships in the same file as the deployment:

deployments:
  - name: bookshop-transform
    entrypoint: flows/transform.py:transform
    work_pool:
      name: k8s-pool
    triggers:
      - type: event
        expect: ["bookshop.orders.loaded"]
        parameters:
          day: "{{ event.payload['day'] }}"

Deploy that, and the automation is created with it. Delete the deployment, and the trigger goes with it. That’s the right lifecycle for a rule whose entire purpose is to start this deployment, and it means the answer to “what causes this pipeline to run?” is in the pipeline’s own file rather than in someone’s browser tab.

Whichever door you use: prefect automation ls should never show you something that isn’t in the repo. An automation nobody remembers creating, that cancels flow runs, is an outage with a countdown on it.

Where hooks end and automations begin

You’ll meet state hooks in two chapters, and they look like they solve this same problem, so plant the flag now.

A hook is Python in your codebase, running in the run’s own process, on the same worker, with the same imports and the same secrets. That makes it perfect for reactions that need the run’s context — drop the staging table this flow created, compute a duration from the flow_run object, call a client you already have open. And it makes it useless for the failure you most need to hear about: if the worker was OOM-killed, the hook was in that worker.

An automation runs on the server, off the event stream, and does not care whether your process is alive. That independence is the entire value. An automation watching for Crashed fires precisely when your on_crashed hook can’t.

The rule: hooks for reactions that belong to the run; automations for reactions that must survive it. A serious setup uses both, and uses them for different things.

Final thoughts

Go back to the opening question. What does Airflow do when a DAG fails? It does what you told it, in the DAG, with the DAG’s own Python, on the DAG’s own infrastructure — which is exactly the infrastructure that might be the thing that failed. The reaction is inside the thing it’s reacting to, and it’s per-DAG, so the hundredth pipeline needs the hundredth copy of the alerting boilerplate, and the one that got it wrong is the one that fails silently.

Prefect pulls the reaction out. Events are emitted by the server, automations are evaluated by the server, and actions are taken by the server — so “page the platform team whenever anything crashes” is one rule, written once, that covers every flow you will ever deploy, including the ones written by people who’ve never heard of it. Then emit_event() lets you push your own domain facts into the same stream, and the same machinery that pages someone at 3 a.m. becomes the machinery that chains your pipelines together on data rather than on cron.

That’s a control plane. It’s the thing you were building by hand out of callbacks and sensors and a Slack webhook you pasted into six DAGs, and the reason it’s worth the switch isn’t that Prefect’s version is prettier. It’s that Prefect’s version keeps working after the process that would have run yours is already dead.

Next: Background Tasks: Your Job Queue Was a Decorator All Along