Retries, Timeouts, and the Failures Worth Repeating

Here is the friendliest chapter in the book. Almost everything in it will feel like home: you already know what retries=3 means, you already know why a delay between attempts is a good idea, and you have already had the argument about whether the number should be three or five. Prefect changes the syntax and adds two or three genuinely better knobs, and that’s it. If you skimmed one chapter of this series, this would be the safe one to skim.

Don’t. Because buried under the familiar surface is a precondition that neither Airflow nor Prefect will ever check for you, that no tutorial states out loud, and that is quietly false in about a third of the pipelines I’ve read: a retry is only correct if the thing you’re retrying is idempotent. retries=3 doesn’t mean “try harder.” It means “I hereby assert that running this function up to four times has the same effect on the world as running it once.” Airflow let you make that assertion by typing a number into a kwarg. So does Prefect. Neither of them will tell you when you’re lying.

So we’ll do the easy material fast, spend real time on the two knobs that let you stop retrying things that will never succeed, be honest about what flow-level retries actually re-execute (the answer surprised me, and I ran it), and finish on the precondition — which is also the on-ramp to the next two chapters, because “make this safe to run twice” is what results, caching, and transactions are all secretly for.

The part that ports cleanly

A flaky extract against an external order feed should try again before it gives up:

from prefect import task

@task(retries=3, retry_delay_seconds=[10, 30, 60])
def extract_orders(day: str) -> list[dict]:
    resp = http_get(f"https://feed.internal/orders?day={day}")
    resp.raise_for_status()
    return resp.json()

retries=3 gives three more attempts after the first failure — four executions in the worst case, the same off-by-one Airflow trained into you. retry_delay_seconds accepts four shapes, and it’s worth knowing all of them because they cover a surprising amount of ground:

A number — every attempt waits that long. retry_delay_seconds=30.
A list — a delay per attempt, as above: 10s, then 30s, then 60s. If the list is shorter than retries, Prefect repeats the final value rather than falling off the end (the engine indexes with min(attempt, len(delays) - 1)), so retries=5, retry_delay_seconds=[10, 60] gives you 10, 60, 60, 60, 60. A list longer than 50 entries is a ValueError — there’s a hard cap.
A callable taking the retry count and returning a list of delays. More on this shortly.
Nothing, in which case the task retries immediately, back-to-back, with no pause at all. This is almost never what you want and it’s the default. A retry with no delay against a service that just 503’d you is not a retry; it’s a second helping of the same failure.

The mapping from Airflow is close to mechanical:

Airflow	Prefect
`retries=3`	`retries=3`
`retry_delay=timedelta(seconds=30)`	`retry_delay_seconds=30`
`retry_exponential_backoff=True`	`retry_delay_seconds=exponential_backoff(...)`
`max_retry_delay=timedelta(minutes=10)`	(no direct equivalent — cap it in your own callable)
`execution_timeout=timedelta(minutes=5)`	`timeout_seconds=300`
`default_args={"retries": 3}`	flow-level defaults, or `@task` on a shared decorator

The real difference isn’t in the table. It’s that in Airflow these values live in default_args, get inherited by every task in the DAG, and are then selectively overridden three files away — so the effective retry policy of any given task is a scavenger hunt. In Prefect the decorator is the configuration surface. The retry policy of extract_orders is written on extract_orders. That sounds like a small thing until the first time you have to answer “why did this task retry eleven times” at 2 a.m. and the answer is on the same screen as the function.

Backoff, and why jitter is not decoration

Fixed delays are fine for a service that’s either up or down. They’re wrong for a service that’s overloaded, because a fixed delay means every client that failed comes back at the same moment with the same enthusiasm. What you want is to back off further each time, and Prefect ships the curve rather than making you write it:

from prefect import task
from prefect.tasks import exponential_backoff

@task(
    retries=4,
    retry_delay_seconds=exponential_backoff(backoff_factor=10),
    retry_jitter_factor=0.5,
)
def extract_orders(day: str) -> list[dict]:
    ...

exponential_backoff is not magic and it’s worth seeing the whole thing, because it’s about six lines: it returns a callable that, given the retry count, produces [backoff_factor * 2**r for r in range(retries)]. With backoff_factor=10 and retries=4 that’s exactly [10, 20, 40, 80] — I ran it, and that’s the list. backoff_factor=3 with retries=5 gives [3, 6, 12, 24, 48]. It also clamps at 50 retries, which you will never hit and shouldn’t want to.

Now the interesting argument. retry_jitter_factor exists to solve a failure mode that has nothing to do with your code and everything to do with the fact that your code has friends.

Picture the actual incident. The orders API falls over at 03:00:00. Two hundred mapped task runs — one per store — all call it within the same second, all get a 503, all fail, all schedule a retry for now + 10s. At 03:00:10, two hundred requests arrive at a service that is still recovering, in a burst tighter than anything it sees in normal traffic. It falls over again. Every one of them backs off to 20s. At 03:00:30, two hundred requests arrive at once. The synchronised retry doesn’t just fail to help — it is now the cause of the continued outage, and it will stay perfectly in lockstep forever, because they all failed together so they all back off together. You have built a distributed metronome that beats the service to death. The literature calls this the thundering herd; the on-call engineer calls it “it recovers for two seconds every thirty.”

The fix is to make the delays disagree. And here is where I have to correct something I’ve seen written a hundred times, including in an earlier draft of this very chapter: retry_jitter_factor does not add up to N% on top of your delay. That’s the intuitive reading and it’s wrong. What Prefect actually does is draw the delay from a clamped Poisson interval centred on your base delay, where the jitter factor is the clamping factor — the upper bound is delay × (1 + factor), and the lower bound is chosen so that the mean stays at your base delay. The spread goes both ways.

I ran twenty thousand draws so you don’t have to. For a base delay of 60 seconds:

retry_jitter_factor=0.2  ->  min 49.5s   mean 60.1s   max 72.0s
retry_jitter_factor=0.5  ->  min 37.8s   mean 60.2s   max 90.0s

Read those numbers carefully, because they tell you how to choose the value. The mean is preserved — jitter doesn’t make your pipeline slower on average, which is the objection people reach for when they leave it off. What changes is the variance, and variance is the entire point: those two hundred retries now land smeared across a 52-second window instead of stacked on one instant. The herd disperses. The service gets a chance to breathe between requests instead of being hit with all of them.

Use 0.2 for a mild smear when you have a handful of concurrent tasks. Use 0.5 or higher when you have a fan-out — a .map() over stores, a hundred parallel API calls, anything where the count of simultaneous retries is large enough to be its own load test. And note that the jitter is applied server-side, by the orchestration rules that schedule the AwaitingRetry state, not by a sleep() in your process — which means it works the same whether your tasks are threads in one flow or pods across a cluster. That’s the version of this feature Airflow never had; retry_exponential_backoff is deterministic, and the jitter is something you were expected to hand-roll into an operator subclass and then forget to.

When the curve isn’t a curve

Exponential is a good default and a bad religion. Sometimes the right schedule is shaped by a fact about the downstream system, not by a formula — and that’s what the callable form of retry_delay_seconds is for. The contract is simple: a function that takes the number of retries and returns a list of delays. That’s the same contract exponential_backoff fulfils internally, so you’re not using an escape hatch, you’re using the same door.

def probe_then_wait_for_the_batch(retries: int) -> list[float]:
    # The vendor's feed rebuilds on a 15-minute cron. One quick probe in case
    # it was a blip; after that there is no point trying again until the
    # next rebuild, so stop hammering and wait for it.
    return [5.0] + [900.0] * (retries - 1)

@task(retries=3, retry_delay_seconds=probe_then_wait_for_the_batch)
def extract_orders(day: str) -> list[dict]:
    ...

That schedule — 5s, 900s, 900s — is not a curve any backoff library would generate, and it’s exactly right for this system: one attempt in case the failure was a dropped packet, then patience aligned to the thing you’re actually waiting for. A capped exponential is the other common one, and Airflow’s max_retry_delay has no Prefect equivalent precisely because you can just write it:

def capped_exponential(retries: int) -> list[float]:
    return [min(600.0, 10 * 2**r) for r in range(retries)]   # never wait > 10 min

One wrinkle worth knowing: the callable is invoked once, at decoration time, with self.retries — the delay list is computed when the task is constructed, not fresh on each failure. So you can’t make the delay depend on the exception, or on how the world looks right now. It’s a schedule, not a reaction. If you need the retry decision to depend on what actually went wrong, that’s a different knob, and it’s the best one in the chapter.

`retry_condition_fn`: not all failures deserve a second chance

Here’s the question retries never ask: should this be retried?

Your task calls an API. The API returns 500. That is a fine thing to retry — the server had a bad moment, and the same request a minute later has a genuinely different chance of succeeding. Now the API returns 404, because you asked for a day the vendor never published. Retrying that is not resilience. It is a machine confidently repeating a question it has already been given a definitive answer to, four times, over ten minutes, before surfacing the error you could have had immediately. And it’s worse than the delay: the alert now fires ten minutes late, the failure looks flaky instead of deterministic (because it “tried several times”), and the next engineer to look at it wastes an hour hunting for a transient cause that never existed.

Same for a ValidationError on a schema you got wrong, a PermissionDenied on a credential that expired, a ZeroDivisionError in your own code. None of these are going to fix themselves in thirty seconds. Retrying them is superstition — a ritual performed in the hope that the universe will relent.

retry_condition_fn is how you stop. It’s a predicate Prefect calls when the task fails, with the signature (task, task_run, state) -> bool, returning True to allow the retry and False to give up now:

from prefect import task

class FeedError(Exception):
    def __init__(self, status: int):
        self.status = status
        super().__init__(f"feed returned {status}")

def is_retryable(task, task_run, state) -> bool:
    """Retry server-side and transport failures. Never retry a client error."""
    exc = state.result(raise_on_failure=False)

    # Transport-level: the request never got a real answer.
    if isinstance(exc, (TimeoutError, ConnectionError)):
        return True

    # Application-level: 5xx is the server's problem, 4xx is ours.
    if isinstance(exc, FeedError):
        return exc.status >= 500 and exc.status != 501   # 501 won't be implemented later

    # Anything we didn't explicitly classify: don't retry. Fail loudly and fast.
    return False

@task(retries=3, retry_delay_seconds=exponential_backoff(10), retry_condition_fn=is_retryable)
def extract_orders(day: str) -> list[dict]:
    ...

Note the shape of is_retryable: it uses state.result(raise_on_failure=False) to get the exception object as a value rather than having it re-raised in the predicate’s face. That’s the same handle you’ll meet properly in the states chapter — a Prefect State carries its failure as data, so a function that inspects failures is just a function, not an exception-handling contortion.

Note also the last line. The default branch is return False, not return True, and that is a deliberate and slightly aggressive choice: classify the failures you know are transient, and treat everything else as a bug until proven otherwise. The opposite default — retry unless you recognise it as fatal — sounds safer and isn’t, because the failures you didn’t anticipate are exactly the ones most likely to be real bugs, and burying a real bug under four retries and a ten-minute delay is how a crisp stack trace becomes a flaky-test folklore.

I ran this to be sure of the semantics, and there are two behaviours worth writing down. First, the predicate genuinely short-circuits: a task with retries=3 whose condition returns False executes its body exactly once and goes straight to Failed. Second — a detail I hadn’t expected — the predicate is only consulted while retries remain. With retries=3 and a condition that always returns True, the body runs four times but is_retryable is called only three. There’s no point asking permission for a retry you weren’t going to get.

The Airflow equivalent of this, if you want to feel good about your move, is either subclassing the operator to catch and re-raise as AirflowFailException (which forces an immediate fail and skips remaining retries) or wrapping the whole body in a try/except that decides between two exception types by hand. It works. It’s also code you write once per operator and copy-paste forever, and it lives inside the task body, tangled up with the business logic, rather than beside it as policy.

Flow-level retries: what actually re-executes

Airflow retries a task. Prefect will also retry the flow — put retries on @flow and a failure anywhere inside re-executes the flow function from the top.

The pitch you’ll read everywhere, including in the first draft of this chapter, is that this is cheap: on the second attempt, tasks that already completed are served from their results and only the unfinished work re-runs, so a flow retry resumes rather than restarts. That’s a lovely story. It’s also, out of the box, false — and I only know that because I ran it.

Here’s the experiment. extract is expensive and always succeeds. load fails on its first two attempts and succeeds on the third. The flow has retries=2.

from prefect import flow, task

@task
def extract(day: str) -> list[int]:
    print("BODY extract")
    return [1, 2, 3]

@task
def load(rows: list[int]) -> None:
    print("BODY load")
    if not warehouse_is_feeling_well():
        raise RuntimeError("warehouse said no")

@flow(retries=2, retry_delay_seconds=30)
def daily(day: str):
    load(extract(day))

What I expected: extract runs once, load runs three times. What actually happened:

extract bodies: 3      load bodies: 3

extract re-executed on every single flow attempt. The expensive task I was promised would be skipped ran three times.

The reason is a chain of defaults that each look reasonable and combine into a trap. A bare @task gets the DEFAULT cache policy — good, and that policy’s key is stable across flow retries. But a bare @task does not persist its result: persist_result is unset, so it falls through to PREFECT_RESULTS_PERSIST_BY_DEFAULT, which ships as False. And a cache key with no stored result behind it is not a cache hit. It’s a key pointing at nothing. Prefect computes the key, looks for a record, finds none, and runs your task — exactly as it should, given that nobody ever wrote the result down.

Turn persistence on — either persist_result=True on the tasks, or PREFECT_RESULTS_PERSIST_BY_DEFAULT=true for the process — and the promise comes true:

extract bodies: 1      load bodies: 3

One extract, three loads. That is the resume-not-restart behaviour, and it is genuinely excellent — it’s the thing an Airflow “clear the DAG run” can never give you, because clearing a DAG run re-runs the tasks and Airflow has no notion of a task’s output being already known. But it is a behaviour you opt into, by making results durable, and the whole of the next chapter is about doing that deliberately instead of by accident.

So the precise statement — the one to carry out of here — is this: a flow retry re-executes the flow function from the top. Every task call in it is evaluated again. A task call whose cache key resolves to a persisted result returns that result without running its body; a task call whose result was never persisted runs again in full. Flow retries don’t skip tasks. Caching skips tasks, and a flow retry is just an ordinary re-execution that caching happens to be very good at.

Which tells you when to reach for the feature. Flow-level retries are the right tool when the failure is about the flow, not about one task — a shared connection that died, a token that expired mid-run, an infrastructure hiccup that poisoned everything downstream. They’re the wrong tool when one task is flaky, because a flow retry re-enters every task’s orchestration machinery to reach the one that broke, and if your results aren’t persisted it re-executes all of them too. Task retries are a scalpel. Flow retries are a restart with a good memory — and only if you gave it one.

Timeouts are a different failure, not a slow success

timeout_seconds catches Airflow engineers off-guard, not because the mechanism is strange but because the category is. A task that runs past its budget doesn’t get politely nudged. It is cancelled, and it goes to a TimedOut state — a failure, with all the consequences of failure.

@task(retries=2, retry_delay_seconds=30, timeout_seconds=30)
def extract_orders(day: str) -> list[dict]:
    ...

A hung TCP connection to the feed — the classic, where the socket is open, no data is coming, and no exception will ever be raised because nothing has gone wrong, it’s just never going to go right — no longer stalls the flow forever. It fails at 30 seconds, and because retries are configured, it comes back for another go with a fresh connection. That’s the composition you want: a timeout behaves exactly like an exception, so everything you’ve built on top of failure keeps working over it.

This is the same idea as Airflow’s execution_timeout, and Prefect adds the obvious sibling: timeout_seconds on @flow, bounding the whole run. Worth having on anything scheduled, because a flow that hangs doesn’t just fail to deliver data — it holds infrastructure, occupies a concurrency slot, and keeps the next scheduled run of the same flow waiting behind it. One hung run at 02:00 quietly becomes a backlog by breakfast. A flow timeout converts an indefinite hang into a bounded, alertable failure, which is the whole job.

Now the honest caveat, because “cancelled” is doing a lot of work in that first paragraph. Prefect enforces a timeout by interrupting your code, and Python is not universally interruptible. Pure-Python work and normal I/O get cut off cleanly. But if your task is blocked inside a C extension that holds the GIL and never checks for signals — a chunky NumPy operation, some database drivers mid-call, a compiled library in a tight loop — the interrupt cannot land until control returns to the interpreter. The timeout is honoured as soon as Python is in a position to honour it, which for a well-behaved task is immediately and for a badly-behaved one is “when it feels like it.”

Two consequences, and they’re not theoretical. A timeout is not a guaranteed kill, so don’t design as though a timed-out task has definitely stopped touching the world. And a timed-out task may have already done part of its work — half a load, some rows inserted, an email sent — and interrupting it doesn’t unwind any of that. Which is precisely the thing the retry sitting next to it is about to walk straight into. Set your timeout with genuine headroom over the p99 (a timeout that fires on a slow-but-fine run is a self-inflicted outage), and if the task has side effects, keep reading, because the next section is about you.

The precondition nobody states

Every retry in this chapter rests on an assumption that no framework can verify and almost no team writes down:

Running this task twice must have the same effect on the world as running it once.

If that’s true, retries are free and you should use them liberally. If it’s false, a retry is not a safety net — it is an amplifier, and you’ve configured it to multiply your worst failure by four.

Look at what the assumption actually rules out:

@task(retries=3)
def load_orders(rows: list[dict]) -> None:
    warehouse.execute("INSERT INTO orders VALUES (...)", rows)   # append

This task is not idempotent, and retries=3 on it is a bug with a friendly face. Attempt one inserts eight thousand rows, then the connection drops on the commit — or, worse, the commit lands and the acknowledgement is lost, which is the same thing from your side. Prefect sees a failure, waits, and retries. Attempt two inserts eight thousand rows again. Now the table has sixteen thousand rows, your revenue number is double, and — this is the cruel part — the flow is green. The retry succeeded. Every dashboard says the pipeline is healthy. You will find out about this from a finance analyst, in a meeting, in three weeks.

Compare that to what would have happened with retries=0: a red flow run, an alert, an engineer, and a table that is missing data rather than lying about it. Missing data is an incident. Wrong data that nobody flagged is a catastrophe with a long fuse. A retry on a non-idempotent task converts a loud failure into a silent corruption, and it does so at exactly the moment you thought you were being careful.

So before you type retries=, ask what the second execution does to the world. There are only a few honest answers:

The task is pure. It reads, it computes, it returns. Retry it freely — this is most extracts, most transforms, and the entire happy path of this feature.
The task’s write is naturally idempotent. A MERGE/upsert on a real primary key, an overwrite of s3://bucket/orders/2026-06-04.parquet, a PUT to a keyed endpoint, a CREATE TABLE ... AS SELECT that replaces the target. Running it twice is running it once. Retry freely.
You can make it idempotent, and you should. Delete-then-insert for the partition inside a database transaction. An idempotency key on the vendor’s API (they almost all support one; use it). A dedupe on load. This is nearly always cheaper than the alternative, and it is the actual engineering work that “make the pipeline reliable” refers to.
It genuinely can’t be made idempotent — a partner’s POST /orders that mints a new order every time, an email you can’t un-send. Then a retry is a decision with a blast radius, and you need a different tool: either retries=0 and a human, or the transaction machinery two chapters from now, which exists precisely so that irreversible side effects happen only on commit, after everything that could veto them already has.

This is not a Prefect problem and I want to be fair: Airflow has exactly the same hole, and the entire genre of “idempotent DAG patterns” blog posts exists because of it. What’s different is that Prefect gives you tools to close it — a cache policy that makes the second execution a no-op, a transaction that makes a half-done group of writes roll back — rather than leaving it as a discipline you enforce with code review and hope. The next two chapters are those tools. Read them as the second half of this one.

Where retries touch the rest of the system

Two seams, so nothing surprises you later.

States. Every word in this chapter is really a statement about state transitions. A failed attempt with retries left doesn’t go to Failed — it goes to AwaitingRetry, which is a name under the Scheduled type, because a task waiting to retry is mechanically just a task scheduled for a moment in the future. It then re-enters as Retrying, runs, and either completes or loops. Failed is only reached when retries are exhausted, and that ordering has a consequence people trip over: an on_failure hook fires once, at the end, not on each failed attempt. If you want to know about every attempt, a state hook is the wrong mechanism. The full model, and the Failed-versus-Crashed split that decides whether a retry is hope or superstition, is the states chapter.

Concurrency. A retry is a scheduled future execution, and when it wakes up it has to acquire a slot like anything else. So a burst of retries against a tag with a concurrency limit doesn’t stampede — it queues, which is a second, quieter defence against the thundering herd, and a good reason to tag every task that leaves your network with the same tag and cap it. It also means your retry delay is a lower bound, not a promise: a task with retry_delay_seconds=30 behind a saturated limit retries in thirty seconds plus however long it waits for a slot. That’s usually what you want and occasionally a surprise. The concurrency chapter has the rest.

Final thoughts

The reason this chapter is the easy one is that retries are the piece of orchestration the industry actually agreed on. Prefect’s version is better in three specific, unglamorous ways — jitter that’s a real distribution instead of a hand-rolled hack, a predicate that lets you refuse to retry a 404, and a flow-level retry that can genuinely resume rather than restart — and those improvements are worth taking. But they’re improvements at the margin. The model is the model you already had.

What’s not the same is what sits underneath. Airflow’s retry is a re-execution and nothing more, because Airflow has no memory of what a task returned; when the retry runs, the task starts from nothing, every time, and “skip the part we already did” is a subgraph you build by hand. Prefect’s retry can be a resume, because Prefect can remember. That’s the whole difference, and you saw it in the experiment above: the same flow, the same retry, one setting apart — and one of them re-ran the expensive extract three times while the other ran it once.

The setting was where the result gets stored. Which is the entire subject of the next chapter, and the reason it comes next: until you know where your return values land, “retry” is a coin flip, “cache” is a rumour, and the idempotency this chapter has been begging you for is something you’ll be enforcing by hand for the rest of your career.

Next: Results and Caching, or: Where Your Return Values Actually Live