Escalation and Errors: Telling Transient From Fatal

How an agent should behave when it can't just proceed — handing off to a human on the right signals (and why sentiment and self-reported confidence aren't among them), asking for clarification instead of guessing, and propagating structured errors through a multi-agent system without either suppressing them or collapsing the whole workflow.

The customer search returns three accounts. A refund request times out. Another customer explicitly asks for a person. Each case interrupts the normal support flow, but each needs a different response: clarify the identity, reconcile the payment outcome, or hand the case over.

Four retry layers lead to failure classification, atomic idempotency, policy-based escalation and structured multi-agent error propagation.

Exam objectives covered here. 5.2 Design effective escalation and ambiguity resolution patterns. 5.3 Implement error propagation strategies across multi-agent systems. Numbering follows the Claude Certified Architect – Foundations Exam Guide, version 1.0, effective July 2026.

What escalation and error propagation are for

Real work is full of moments where the right move is not to proceed. The request is outside what the agent is allowed to decide, a tool came back broken, or the customer is asking for something the policy does not cover. Two disciplines cover those moments. Escalation hands the problem to a human when the agent should not or cannot resolve it. Error propagation governs how a failure travels through a multi-agent system without being either hidden or allowed to bring everything down.

These are the Customer Support and Multi-Agent Research scenarios under stress, and the underlying structured-error mechanics build on Domain 2. A good deal of this chapter is design judgment about when to do things, and that judgment is stated as such. But the mechanics underneath it are not judgment at all. They are a class hierarchy, a retry policy and a set of defaults, all sitting on disk in the installed package. Every one of them is introspected below rather than asserted.

Why it matters, and the cost of getting it wrong

The entire value of an agent is that it resolves things on its own. One that escalates every hard-looking case is an expensive router to a human queue: you have automated nothing, and you have taught your users that the agent is a speed bump on the way to a person. Define the conditions that require a handoff so routine work can continue within the agent’s authority.

Proceeding without enough evidence can affect the wrong account or report success on a failed search. Review both kinds of error: unnecessary escalation and unsupported action. A useful policy distinguishes them by the facts of the case.

The spectrum of “I can’t just proceed”

Choose a response from the evidence available and the agent’s authority:

  • Recover locally. The failure is transient — a timeout, a flaky call. The agent retries and proceeds. No human, no propagation.
  • Ask for clarification. The request is resolvable but ambiguous. The agent has the capability, it just does not yet know which entity or action is meant. It asks one targeted question and continues.
  • Escalate to a human. The situation is beyond the agent’s authority or coverage — an explicit request for a person, a policy gap, or an inability to make progress. The agent hands off.
  • Propagate a structured error. In a multi-agent system, the failure is something this agent cannot resolve but a coordinator might. It reports the failure richly, upward, rather than swallowing it or dying on it.

There is a fifth point that most treatments of this topic leave out, and it is a member of the API’s own closed set. The model can decline. refusal is a stop_reason value in anthropic 0.120.0, alongside end_turn, tool_use and the rest. A refusal is not an error and not an escalation — it is the model exercising a boundary. A loop that treats every non-end_turn stop reason as a failure will retry it forever. Handle refusal explicitly: surface it, do not retry it, and decide by policy whether it routes to a human.

The recurring mistake across all five is collapsing the spectrum: treating a clarification as an escalation, a transient error as a fatal one, or a real failure as a success. Everything else in this chapter is a way of telling those apart.

When to escalate

The support policy should name the conditions that require a human:

  • The customer explicitly asks for a human. This is the clearest trigger, and it is immediate. You honour it; you do not first try to resolve the issue yourself.
  • Policy is ambiguous or silent on the customer’s specific request, not merely when the case is complex. A policy that addresses own-site price adjustments but says nothing about competitor price matching is a gap. A gap is an escalation trigger even if the request itself is simple. This is the subtle one: escalation is about policy coverage, not difficulty.
  • The agent cannot make meaningful progress: it is stuck, not just slow.
  • The message contains a category your policy routes on regardless of the case. Threats of violence, self-harm, an allegation of fraud, a stated legal or regulatory demand, harassment of the agent or of a named person. These are not judgment calls about difficulty. They are enumerated categories with a named destination, and several route to a specialist queue rather than to general support.

Emotional intensity and self-reported confidence do not establish whether the agent has authority to resolve a case. An angry customer may have an ordinary in-policy request; a calm customer may be asking for an exception. Route on policy coverage, progress, and explicit requests.

Hold that rule at exactly the width it was written, because there is a real distinction underneath it that the flat version loses. The claim is that emotional intensity does not predict whether a case needs a human. It is not a claim that the content of a message is irrelevant. “This is unacceptable and I am furious” is intensity: frustration attached to a request the agent can resolve. “I will be reporting this to my lawyer,” “you people have stolen from me,” “I do not want to be here any more” are content. Each names a category with its own handling path, and a specialist queue exists for a reason. The difference is testable rather than a matter of taste — the first is a tone a classifier scores on a continuum, the second is a category your policy enumerates. Write the categories out, route them deterministically, and log which one fired. Route on the score and you are back to escalating capital letters.

Both errors are real, and they are not symmetric in cost. A system that escalates every frustrated customer wastes human capacity on the far larger population, which is expensive. A system that treats an explicit safety or legal signal as “an angry customer, in policy, resolve it” has failed at something no efficiency argument covers. And note what this does not touch: the confidence rule below is unchanged. A deterministic category match is not a confidence score, and adding safety routing is not a licence to escalate on the model’s own uncertainty.

Self-reported confidence deserves its own sentence, because this book uses it three times and they are not the same use. The rule, stated once and held to: uncalibrated self-reported confidence is noise; calibrated against a labeled set it is a legitimate batch-review routing signal; it is never an escalation trigger for a live case. Chapter 11 used it for the second of those, routing findings in a review pass. Chapter 14 is about earning the calibration that makes the second use legitimate. This chapter forbids the third. The reason the live case is different is that calibration is a population property. Knowing that “0.9” means 90% across ten thousand extractions tells you how to allocate reviewer hours. It tells you nothing useful about whether this caller needs a person right now.

Distinguish a request for a person from frustration alone. Honor an explicit human request immediately. For a frustrated customer with an in-policy issue, acknowledge the frustration and offer to resolve it; if they ask for a person, hand the case over.

You implement this with explicit escalation criteria plus few-shot examples in the system prompt, showing the model when to escalate versus resolve. It is exactly the criteria-plus-examples pattern from Domain 4. Vague instructions (“escalate difficult cases”) produce inconsistent escalation; concrete criteria with worked examples produce calibrated escalation.

Ambiguity: clarify, don’t guess

A related reliability behavior sits one notch down the spectrum from escalation. When a tool returns multiple matches — a customer search that hits three accounts — the agent must ask for an additional identifier rather than heuristically selecting one. Picking “probably the most recent” or “probably the first” is a guess, and it will sometimes act on the wrong customer’s account. For a support agent that is a serious error.

The general principle recurs across the exam: when the agent is genuinely uncertain about which entity or action is meant, surfacing the ambiguity beats resolving it by heuristic. Note what separates this from escalation. A clarifying question keeps the agent in control of the case and costs one turn. An escalation hands the case away and costs a human. Reaching for the second when the first would do is the expensive mistake — reaching for a heuristic when the first would do is the dangerous one.

The tool design that supports this is Domain 2’s. A search that returns three matches has succeeded; three results is a valid outcome, not an error. It is the agent’s job to notice that a single-subject action cannot proceed on a three-element result. That distinction only exists if the tool returns the matches rather than silently picking one for you — one more reason a tool should never disambiguate on the caller’s behalf.

Telling transient from fatal: the actual taxonomy

“Recover locally, the failure is transient” is useless advice until you can tell which failures are transient. The anthropic SDK answers that question precisely, because the answer is a class hierarchy with status codes attached. Here it is, introspected from the installed package:

import anthropic
for n in ["AnthropicError", "APIError", "APIStatusError", "APIConnectionError",
          "APITimeoutError", "BadRequestError", "AuthenticationError",
          "PermissionDeniedError", "NotFoundError", "ConflictError",
          "RequestTooLargeError", "UnprocessableEntityError", "RateLimitError",
          "InternalServerError", "OverloadedError"]:
    c = getattr(anthropic, n)
    print(f"{n:26} status={getattr(c, 'status_code', '-')!s:5} "
          f"mro={[b.__name__ for b in c.__mro__[1:4]]}")
AnthropicError             status=-     mro=['Exception', 'BaseException', 'object']
APIError                   status=-     mro=['AnthropicError', 'Exception', 'BaseException']
APIStatusError             status=-     mro=['APIError', 'AnthropicError', 'Exception']
APIConnectionError         status=-     mro=['APIError', 'AnthropicError', 'Exception']
APITimeoutError            status=-     mro=['APIConnectionError', 'APIError', 'AnthropicError']
BadRequestError            status=400   mro=['APIStatusError', 'APIError', 'AnthropicError']
AuthenticationError        status=401   mro=['APIStatusError', 'APIError', 'AnthropicError']
PermissionDeniedError      status=403   mro=['APIStatusError', 'APIError', 'AnthropicError']
NotFoundError              status=404   mro=['APIStatusError', 'APIError', 'AnthropicError']
ConflictError              status=409   mro=['APIStatusError', 'APIError', 'AnthropicError']
RequestTooLargeError       status=413   mro=['APIStatusError', 'APIError', 'AnthropicError']
UnprocessableEntityError   status=422   mro=['APIStatusError', 'APIError', 'AnthropicError']
RateLimitError             status=429   mro=['APIStatusError', 'APIError', 'AnthropicError']
InternalServerError        status=-     mro=['APIStatusError', 'APIError', 'AnthropicError']
OverloadedError            status=529   mro=['APIStatusError', 'APIError', 'AnthropicError']

Everything descends from AnthropicError, so one except AnthropicError catches the lot. Below it the tree forks in exactly the way the spectrum needs.

APIConnectionError and its subclass APITimeoutError mean you never got an answer. The network failed or the read deadline passed. Nothing about the request itself was wrong, so the condition is transient and the same bytes may well succeed on a second attempt.

Read that sentence again, because there is a much sharper claim people hear in it. “I got no answer” is not the same as “it did not happen.” A timeout tells you the response was lost; it says nothing about whether the server processed the request before losing it. For a read — a messages.create, a lookup_order — the distinction does not matter, and retrying is free. For anything that changes state, a timeout is a third outcome, and it deserves its own name: not success, not failure, but unknown. The refund may have gone through. Treat unknown as failure and you refund twice; treat it as success and you strand a customer. The correct move is to find out before you replay — reconcile against the operation’s own status, keyed by something stable, and only then decide. That is the entire reason idempotency gets a section of its own later in this chapter. It is also why the key has to be minted before the first attempt rather than after a failure.

APIStatusError and its subclasses mean the server answered, and the status code is the verdict. Split them into two groups and the “transient or fatal” question stops being a judgment call:

  • Fatal, in the sense that an identical retry cannot help. BadRequestError (400) and UnprocessableEntityError (422) mean the request was malformed; send the same bytes again and get the same answer. AuthenticationError (401) and PermissionDeniedError (403) mean credentials or entitlement, which retrying will not acquire. NotFoundError (404) means the thing is not there. RequestTooLargeError (413) means the payload is over the limit, and the fix is the previous chapter’s, which is to send less. These are the four-category validation and permission errors from chapter 4, wearing HTTP numbers.
  • Transient, in the sense that the same request may well succeed later. RateLimitError (429) means slow down. InternalServerError (any 5xx) and OverloadedError (529) mean the far side is having a bad moment. ConflictError (409) means a lock contention that should clear.

OverloadedError at 529 is the one worth memorizing, because it is a non-standard code that people mistake for something fatal. It is capacity pressure — precisely the case for backing off rather than failing the user.

That gives you a handler you can actually write:

import anthropic

FATAL = (anthropic.BadRequestError, anthropic.AuthenticationError,
         anthropic.PermissionDeniedError, anthropic.NotFoundError,
         anthropic.RequestTooLargeError, anthropic.UnprocessableEntityError)

try:
    msg = client.messages.create(...)
except FATAL as e:
    # Retrying is pure waste. Fix the request, or escalate.
    log.exception("model call failed")              # internals stay in the log
    report_structured_error(kind="fatal", status=e.status_code,
                            code="model_request_rejected",
                            diagnostic_ref=incident_id())   # opaque handle
except (anthropic.RateLimitError, anthropic.InternalServerError,
        anthropic.OverloadedError, anthropic.APIConnectionError) as e:
    # Transient. But read the next section before you write a retry here.
    degrade_or_defer(e)

Note what is not in that call. str(e) does not belong in anything a model or another agent will read. Exception text is written for an operator. It carries URLs, header fragments, request ids, file paths, sometimes the offending payload, and on a bad day a credential that was in scope. Everything you propagate becomes context for a downstream agent, and everything in context can be summarized, quoted back to a customer, or written to a scratchpad. It is also unstable — the message wording is not an API, so any coordinator that branches on its substring breaks on an SDK upgrade. Emit a stable code the caller can branch on, a message safe to show a human, and an opaque reference that lets an engineer find the full exception in your logs. Prove it with a test: plant a canary secret in the environment and assert it never appears in propagated context.

The class names, the status codes and the inheritance chains above are introspected from anthropic 0.120.0. Which errors a given production workload actually encounters is a property of that workload and was not measured here.

And treat the code as evidence rather than as the whole decision. The split above is a strong default for a model API call. It is exactly what the SDK’s own policy implements, so learn it in that form. But the same number carries different meanings on different endpoints, and a rule expressed purely as a status range will eventually be wrong on yours:

SignalDefault readingWhen the default is wrong
404Fatal — the thing is not thereA record written moments ago on an eventually-consistent store may 404 briefly. A short bounded retry is correct there and nowhere else.
409Transient — a lock that should clearA permanent conflict (an idempotency key reused for different arguments, an edit against a stale version) never clears. Retrying it is a loop.
429Transient — back offHonour Retry-After when it is present; a quota exhausted for the calendar month is not going to clear inside your deadline.
400Fatal — the request is malformedStill fatal, but check why: an oversize request is a 413 you can fix by sending less, which is a repair, not a retry.
TimeoutTransientFor a state-changing call the outcome is unknown, so reconcile before replaying.

Three questions decide it, in order. Is the outcome known? If not, reconcile first, whatever the code says. Is the operation idempotent? If not, a retry needs a key before it needs a policy. Does the server say anything? Retry-After and the SDK’s x-should-retry header outrank your table. The status code is the fourth input, not the first.

The retries you already have

Before adding another attempt in the exception handler, look at what the client already does. Two things here are usually a surprise.

import anthropic
client = anthropic.Anthropic()
print(client.max_retries)   # 2
print(client.timeout)       # Timeout(connect=5.0, read=600, write=600, pool=600)

max_retries defaults to 2. So a call that raises RateLimitError in your code has already been attempted three times, with waits in between, before the exception ever reached you. Wrap that in your own “retry three times” loop and you have not built a three-attempt policy. You have built a nine-attempt one — invisible in your own logs, and aimed at a service that just told you it is rate limiting you. Multiplying your pressure against a 429 is the exact wrong response to a 429.

The policy the SDK applies is not guesswork either. It is in _base_client.py and it is short:

_should_retry(response):
    x-should-retry header, if present, wins outright ("true" / "false")
    408  -> retry   (request timeout)
    409  -> retry   (lock timeout)
    429  -> retry   (rate limit)
    >=500 -> retry
    otherwise -> do not retry

The status rules include 408 and 409 as explicit retry cases, alongside 429 and server errors. The source uses INITIAL_RETRY_DELAY = 0.5 and MAX_RETRY_DELAY = 8.0, with delay min(0.5 * 2**n, 8.0) multiplied by 1 - 0.25 * random(). A retry-after value between 0 and 60 seconds takes precedence over that backoff.

The second surprise is the timeout. The default read timeout is 600 seconds. Ten minutes. That is a sensible ceiling for a long batch-style generation and an absurd one for a customer support agent where a person is watching a spinner. Nothing will tell you about it — the call simply sits there. Set it deliberately, per workload:

client = anthropic.Anthropic(timeout=30.0, max_retries=1)   # interactive support agent

And note the interaction: your effective worst case is roughly timeout * (max_retries + 1) plus backoff. At the defaults, a single unlucky call can occupy thirty minutes before it raises.

So what should you add on top? Not attempts. Add the three things the SDK deliberately does not know about:

  • A deadline. The SDK bounds attempts, not wall-clock. A user-facing turn should carry an absolute “answer by” time and give up against that, whatever the attempt count says.
  • A retry budget. Cap retries as a fraction of total requests across the whole service rather than per call. Per-call limits still allow every caller to triple its load simultaneously during an incident, which is how a brief overload becomes a long one.
  • A circuit breaker. After N consecutive transient failures, stop calling for a cooling period and fail fast. Two SDK retries plus your own retry plus every concurrent request doing the same is a stampede aimed at a service that is already down.

Say your middleware raises its own exceptions and you want them retried under the SDK’s policy rather than your own. anthropic.RetryableError is a public class that exists for exactly that. It subclasses AnthropicError, and the retry check walks each exception’s __cause__ chain, so raise MyError(...) from retryable preserves retries.

Four layers, and only one of them is the SDK’s

Everything above is about one layer, and the commonest reasoning error in this whole chapter is to assume it covers the others. client.max_retries = 2 governs HTTP calls the anthropic client makes to the Anthropic API. It has no opinion whatsoever about your refund service. Four distinct layers can produce an attempt, they compose multiplicatively, and each is configured somewhere else:

LayerWhat it retriesWhere you set it
1 · Model API transportThe HTTP request to Anthropicclient.max_retries, default 2
2 · The agent loopReissues a tool call when the model asks again — under the Agent SDK, autonomouslyPrompt, max_turns, tool error contract
3 · Tool transportYour tool client’s own HTTP calls to the refund serviceYour HTTP client, or the MCP server’s
4 · Business operationNothing. This is where a repeat must be absorbedIdempotency key in the refund service

Trace one refund through them. The model emits a tool_use for process_refund, and your handler calls the refund service. The service’s client retries a 503 twice (layer 3), and the call still fails, so the handler returns a tool_result marked as an error. The model reads it and, because your description invited a retry, calls process_refund again (layer 2). Layer 1 was never involved — the Anthropic client’s max_retries did not fire once, because the failing call was not to Anthropic. So tuning max_retries down to zero would not have prevented a single duplicate attempt on the money.

The practical consequences are worth stating flatly. Retry at exactly one layer per hop, and prefer the one closest to the failure, because it knows the most about it. Cap the total, not each layer: two SDK attempts times two tool-client attempts times two agent reissues is eight calls, and nobody wrote the number eight anywhere. Label every attempt in your telemetry with which layer produced it, or an incident becomes an argument. And make layer 4 safe regardless, because layer 2 is the one you cannot fully control: the model decides to call the tool again, and no client setting stops it.

Degrading instead of failing

A circuit breaker only helps if there is something behind it. Graceful degradation is the answer to “the breaker is open, now what,” and it has a rough order of preference:

  1. A cached answer. If the same question was answered recently and the answer is not time-sensitive, serve it and say so.
  2. A smaller or different model. The Claude Code CLI exposes --fallback-model <model> for automatic fallback, and under the SDK ClaudeSDKClient.set_model lets you switch mid-session. A cheaper model answering is better than no answer, provided the workload tolerates the quality change. Deciding whether yours does is a measurement question, not an assumption.
  3. A reduced-capability response. Answer the part you can. Look up the order without proposing the refund.
  4. An honest canned response plus a handoff. “I cannot reach the order system right now. I have opened a ticket with your details.” This is a real outcome, not a failure — and it is enormously better than a hang.

The ordering matters more than the list. Every step down should be visible, both to the user and in your telemetry. A silently degraded agent that keeps returning worse answers is exactly the “proceeds when it should not” failure this chapter opened with.

Both of the first two steps have conditions, and a flag that makes them one line of configuration is exactly what hides those conditions.

A cached answer is a different answer unless you key it correctly. Cache on the whole of what determined the original response, not on the question text. That means the tenant, the authenticated principal and their entitlements, the full input, and the version of the policy or prompt that produced it. Key on the question alone and you will eventually serve one customer’s order status to another, which is a data-protection incident rather than a degraded response. Then attach a freshness contract. Every cached response carries the time it was computed and is stamped as stale when it is served, both to the user (“as of 09:41 today”) and in your logs. And decide in advance which answers may never be stale — a balance, an eligibility decision, anything a person will act on financially — and let those fail honestly instead. The failure mode here is that stale-serving conceals the outage: your error rate looks fine, your dashboards are green, and your customers are reading yesterday.

A fallback model is a different system, not the same system running cheaper. Before one is eligible it has to pass the same contract and safety suite as the primary, on the same evals, per segment. Does it honour your tool schemas? Does it support extended thinking if your loop relays thinking blocks? Does its structured output validate, does it refuse where you need it to refuse, and is it approved for the regions and data your workload runs under? A model that fails any of those is not a degraded answer — it is a wrong answer arriving faster. Then make the switch observable and reversible. Route by an explicit versioned rule rather than an ambient flag, and record on every response which model actually served it and why the fallback fired. Hold the fallback to its own quality threshold, so you can tell when degradation stopped being acceptable. The ResultMessage.model_usage map from the previous chapter is keyed by the model that really served the request, which is precisely the field this needs.

The status code hiding in a successful result

Under the Agent SDK the error surface looks different, and there is a trap in it worth knowing on sight. ResultMessage carries both is_error and subtype, and they can disagree. From the installed package’s own comment on the field:

# HTTP status code (e.g. 429, 500, 529) of the failing API call when
# ``is_error`` is True and ``subtype`` is "success"; None otherwise.
api_error_status: int | None = None

A success subtype can accompany is_error=True. Check both fields, and read api_error_status, the HTTP status of the failing API call, when it is set. The result also carries errors: list[str] | None and terminal_reason: str | None, the latter distinguishing a normal completion from "max_turns" or an interrupted "aborted_streaming".

Seeing the limit before you hit it

Reacting to a 429 is the late option. The Agent SDK emits rate limit information proactively, and the type names are worth knowing because they encode a warning tier:

import claude_agent_sdk as sdk, typing
print(typing.get_args(sdk.RateLimitStatus))
# ('allowed', 'allowed_warning', 'rejected')
print(typing.get_args(sdk.RateLimitType))
# ('five_hour', 'seven_day', 'seven_day_opus', 'seven_day_sonnet', 'overage')
print(list(typing.get_type_hints(sdk.RateLimitInfo)))
# ['status', 'resets_at', 'rate_limit_type', 'utilization',
#  'overage_status', 'overage_resets_at', 'overage_disabled_reason', 'raw']

allowed_warning is the whole point. It is the state between fine and rejected, and it arrives on a RateLimitEvent before anything fails. With utilization and resets_at alongside it, you can shed load, defer batch work, or warn an operator while requests are still succeeding. Handling rate limits only in an exception handler means you never see this tier. The type shapes here are introspected; the emission cadence of these events under real load was not exercised for this book.

Idempotency, or how retries move money twice

Suppose process_refund reaches the refund service, which commits the debit, and then the response is lost. The tool’s HTTP client may retry. If the handler returns an error, the model may request the tool again. Those are the tool-transport and agent-loop layers described above, and the Anthropic client’s retry setting governs neither of them. So without deduplication, more than one attempt can move money even though every caller is following its own retry policy. A missing response does not establish that the operation was never executed, which means the handler has to recognise an operation it has already performed before it allows another attempt.

The fix is an idempotency key: a value that identifies the intended operation, not the attempt. Two things decide whether it works, and both are easy to get wrong.

The first is where the key comes from. Mint it once, upstream, in the code that decided the refund — do not compute it inside the tool. Approving a refund on a case writes an authorization row, and that row’s id is the operation id. Every attempt afterwards presents it, so the SDK’s retries, your wrapper’s retries and a tool call the agent re-issues all carry the same key. The model neither generates the key nor sees it; the wrapper looks it up from the authorization the case already holds.

Deriving the key inside the tool from the arguments is the trap, and free text is the sharpest edge on it. Hash the reason into the key and two attempts at the same refund with differently-worded reasons produce different keys, so the guard never fires and the money moves twice. The reason still matters, but as something you check, not something you key on. Store a fingerprint of the fields that define the money movement, and compare it on replay.

The second is how the check is enforced. A get, then an if, then a create is three steps with gaps between them, and two concurrent attempts fit through those gaps side by side. Both read no prior record, both pass the if, both refund. The check has to be a unique constraint committed in the same transaction as the side effect. The loser of a race then loses at the database, rather than at a branch it already ran past.

CREATE TABLE refund_operations (
    operation_id  TEXT PRIMARY KEY,   -- the caller's key. Never derived from free text.
    fingerprint   TEXT NOT NULL,      -- hash of the money-moving fields
    refund_id     TEXT NOT NULL,
    amount_cents  INTEGER NOT NULL,
    created_at    TIMESTAMPTZ NOT NULL DEFAULT now()
);
import hashlib, json

def fingerprint(args) -> str:
    # Only the fields that define the money movement. Not the reason text.
    body = json.dumps({"order_id": args["order_id"],
                       "amount_cents": args["amount_cents"]}, sort_keys=True)
    return hashlib.sha256(body.encode()).hexdigest()

def replay(row, status):
    return {"content": [{"type": "text", "text": json.dumps(
        {"refund_id": row.refund_id, "status": status,
         "amount_cents": row.amount_cents})}]}

async def process_refund(args):
    key, fp = args["operation_id"], fingerprint(args)

    prior = await refunds.get(key)
    if prior is not None:
        if prior.fingerprint != fp:
            return {"content": [{"type": "text", "text": json.dumps(
                {"error": "operation_id already used for a different refund",
                 "isRetryable": False})}], "is_error": True}
        return replay(prior, "already_processed")

    try:
        async with db.transaction():          # ONE transaction, both writes
            refund = await ledger.debit(args["order_id"], args["amount_cents"])
            await refunds.insert(operation_id=key, fingerprint=fp,
                                 refund_id=refund.id,
                                 amount_cents=args["amount_cents"])
    except UniqueViolation:
        # A concurrent attempt committed first. This transaction — the debit
        # included — rolled back. Read the winner's row, and apply the SAME
        # conflict check as the sequential path: the race does not make a
        # mismatched request safe to replay.
        won = await refunds.get(key)
        if won.fingerprint != fp:
            return {"content": [{"type": "text", "text": json.dumps(
                {"error": "operation_id already used for a different refund",
                 "isRetryable": False})}], "is_error": True}
        return replay(won, "already_processed")

    return replay(await refunds.get(key), "processed")

Verification of the error flag: both conflict branches were passed through the installed Python SDK’s MCP wrapper and returned isError=True. The database was stubbed; this checks result marshalling, not transaction isolation or concurrent debit behavior.

Both the sequential replay path and the concurrent-race path check the fingerprint. A mismatched request is rejected with the Python handler key is_error; the SDK serializes it as MCP isError, as chapter 4 demonstrates. Otherwise, a reused operation id with a different amount could receive the winner’s receipt or lose its error flag.

That gives the tool a property the earlier shape did not have. At most one debit per operation id, under concurrency, enforced by the database rather than by a branch. The old version was safe only when the two attempts happened to be far enough apart, which is exactly the condition a retry storm removes.

Three parts of that handler carry the guarantee, and they only work together.

The key names an authorization, not an argument list. A UUID generated inside the tool changes on every retry and buys you nothing. A hash of the arguments buys you the wrong thing. It turns a reworded retry into a second refund, and a genuine second refund on the same order for the same amount into a silent replay of the first. An id minted once per approved refund has neither failure.

Conflict semantics are explicit, all three cases. Key unseen: execute, and commit the debit and the key row together. Key seen with a matching fingerprint: replay the stored result, move no money. Key seen with a different fingerprint: refuse, do not execute, and mark it isRetryable: false — the caller reused an operation id, which is a bug in the caller, not a transient fault.

Replay returns the stored outcome. A retry with the same operation id and fingerprint should return the first refund’s receipt. Check the refund client’s retry rules before choosing an HTTP status for replays: if it retries 409, a conflict response creates unnecessary attempts. The Anthropic client’s retry list describes calls to Anthropic, not calls to the refund service. Reporting "status": "already_processed" lets the agent explain that the existing refund is in flight without announcing a second payment.

One limit worth naming. This works because the ledger and the key table are in the same database, so one transaction covers both. When the side effect lives in a payment provider you cannot enlist, no local transaction can make the pair atomic. Reserve the key in its own committed row before the call, pass the same key to the provider as its idempotency key, and record the outcome afterwards. That moves the guarantee to the provider, and leaves you a reconciliation job for reservations that never got an outcome. The pattern here is the one the code shows; the distributed case is described rather than run.

The general rule the exam is testing underneath this: any tool that changes state outside the conversation must be safe to call twice. Refunds, orders, emails, tickets, deployments. If it is not, do not let it be retried, by you or by the agent, and say so in the tool’s own error contract with isRetryable: false. The one thing you may not do is leave it retryable and unprotected, which is the state this chapter’s advice would otherwise have left you in.

Error propagation across agents

The rightmost point on the spectrum is how errors travel through a multi-agent system. It has two named anti-patterns you must recognize on sight, because they are opposite failures:

  • Silently suppressing errors, returning empty results as if the query succeeded. Now the coordinator thinks there was genuinely nothing to find, when actually the search failed, and it builds a report on a false premise.
  • Terminating the entire workflow on a single failure, where one subagent’s timeout kills the whole research task, discarding the good results the other subagents produced.

When a subagent cannot recover, return the failure type, attempts, partial results, and permitted alternatives. This example preserves two findings while identifying the missing source:

{
  "status": "partial_failure",
  "failure_type": "upstream_timeout",
  "attempted": {
    "subtopic": "competitor pricing 2024-2026",
    "sources_queried": ["sec_filings", "industry_reports", "news_archive"],
    "failed_source": "industry_reports",
    "attempts": 3,
    "last_error": "read timeout after 30s"
  },
  "partial_results": [
    {"claim": "Competitor A raised list price 8% in Q3 2025",
     "source": "sec_filings:10-Q-2025-Q3", "confidence": "high"},
    {"claim": "Competitor B introduced a usage tier in Jan 2026",
     "source": "news_archive:2026-01-14", "confidence": "medium"}
  ],
  "coverage_gap": "No industry-analyst view on 2026 pricing. Findings cover filings and news only.",
  "alternatives": [
    "Retry industry_reports with a narrower date range",
    "Substitute analyst_summaries source",
    "Proceed and annotate the gap"
  ]
}

The coordinator can see which source failed, what was already attempted, and which findings remain usable. It can assess the proposed alternatives against the assignment’s failure policy. A generic “search unavailable” status omits that evidence.

The access-failure-versus-empty-result distinction from Domain 2 is load-bearing here. The coordinator must be able to tell a timeout — which might warrant a retry — from a valid empty result, a successful query with no matches. It responds to the two completely differently. And subagents should do local recovery for transient failures, retrying the timeout themselves and propagating only what they genuinely cannot resolve. That is the “recover locally” point on the spectrum, done at the subagent level before anything gets propagated at all.

The reliability capstone of this task concerns synthesis. When synthesis combines findings, it should carry coverage annotations, marking which conclusions are well-supported versus which topic areas have gaps because a source was unavailable. A report that silently omits the topics its failed subagent was supposed to cover looks complete — and is not. A report that annotates the gap is honest about what it does and does not know. That honesty about coverage is itself a reliability property, and it flows directly into provenance and uncertainty in the next chapter.

Enforcing the policy instead of requesting it

Everything above is a policy, and a policy that lives only in a system prompt is a request. The SDK gives you a place to make it structural. PostToolUseFailure is one of the ten hook events, and its input carries what a propagation policy needs:

import typing
from claude_agent_sdk import PostToolUseFailureHookInput
print(list(typing.get_type_hints(PostToolUseFailureHookInput)))
# ['session_id', 'transcript_path', 'cwd', 'permission_mode', 'agent_id',
#  'agent_type', 'hook_event_name', 'tool_name', 'tool_input',
#  'tool_use_id', 'error', 'is_interrupt']

tool_name, tool_input and error together are the failure, and agent_id and agent_type tell you which subagent it happened in. is_interrupt distinguishes a cancellation from a genuine fault — exactly the distinction a naive handler gets wrong, by counting a user’s Ctrl-C as a tool failure. Attach a hook here and you get a single choke point for every tool failure in the system. Each one can be logged with a consistent shape, counted toward a circuit breaker, or rewritten into the structured error above before the model ever sees it. Run it and two things show up that the type alone does not tell you. Attaching hooks to both PostToolUse and PostToolUseFailure, then asking the agent to cat a file that does not exist:

PostToolUse fired for    : []
PostToolUseFailure fired : 1 time(s)
   tool_name   : Bash
   is_interrupt: False
   error       : Exit code 1
                 cat: /tmp/definitely-not-here-8471.txt: No such file or directory

First, the two events are mutually exclusive. A failing call fires PostToolUseFailure and does not also fire PostToolUse, so a redaction or logging hook attached only to the success event sees nothing on the path where you most want a record. Attach to both.

Second, a non-zero shell exit counts as a tool failure rather than a successful call that returned error text — the error field carries the exit code and the stderr together. That is the opposite of the is_error convention in chapter 4, where a tool reports its own business failure inside a successful result. The distinction is worth holding: a tool that ran and disagreed with you is is_error; a tool that could not run is PostToolUseFailure.

The live input also carried more than the introspected TypedDict shows — duration_ms, effort and prompt_id were all present alongside the documented fields.

Final thoughts

What to carry forward.

  • 400, 401, 403, 404, 413 and 422 will not improve on a retry; 408, 409, 429, 5xx and 529 may. Idempotency, an unknown outcome and a Retry-After header all outrank the code.
  • A timeout means the answer was lost, not that the work did not happen.
  • The SDK already retries twice with jittered backoff and honours retry-after. max_retries governs only the model API hop; the agent loop, your tool client and the business operation are three further layers. Cut the 600-second read timeout, and add deadlines, budgets and breakers rather than attempts.
  • Degrade deliberately: a cached answer keyed by tenant and labelled stale, a fallback model that passed the same suite. Propagate a stable code and an opaque reference, not str(e).
  • Every state-changing tool is idempotent, because at-least-once is what retries give you. Escalate on an explicit request for a human, a policy gap, an inability to progress, or an enumerated safety, legal or abuse category; never on sentiment or self-reported confidence.

For the three-customer search, obtain the identifier that makes the intended account unambiguous. For the timed-out refund, look up the operation before repeating it. For the explicit human request, hand over the case with enough evidence that the customer does not have to start again.

A coordinator needs the same clarity from its specialists: what succeeded, what remains unknown, and what recovery has already been tried. That record lets it follow the failure policy without hiding a gap or discarding useful work.

Next: human review, confidence, and provenance — calibrating when to trust automation, and preserving where each claim came from.

Comments