Assume Everything Fails: Timeouts, Retries, Limits, and Breakers

The patterns that keep a Go service standing when its dependencies don't — context deadlines on every call, retries with exponential backoff and jitter, rate limiting with x/time/rate, a circuit breaker, and panic recovery — each one built and run. Compiled and run against Go 1.26.5.

Everything you have built so far assumed the world cooperates: the database answers, the upstream API returns, the network delivers your bytes. Production makes no such promise. A dependency will be slow, then unavailable, then flap back and forth in the worst possible rhythm. Your job is not to prevent that — you can’t — but to make sure that when it happens, your service degrades instead of dying, sheds load instead of amplifying it, and comes back on its own when the trouble passes.

This final chapter is a small toolkit for a hostile production: timeouts on every outbound call, retries that back off instead of hammering, rate limits that protect you and the things you depend on, a circuit breaker that stops beating a dead service, and panic recovery so one bad request never takes the process with it. Each pattern below was built and run against Go 1.26.5, and the outputs are real.

Timeouts everywhere

The first rule is the least glamorous: every call that leaves your process needs a deadline. A call without one waits forever by default, and “forever” is how a single slow dependency turns into a pile of stuck goroutines, then exhausted connections, then a dead service. In Go the deadline rides on a context, and every well-behaved client — HTTP, database, gRPC — takes one.

ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()

req, _ := http.NewRequestWithContext(ctx, http.MethodGet, upstream.URL, nil)
resp, err := http.DefaultClient.Do(req)

Point that at an upstream that takes 300ms to answer, and the call gives up at 100ms instead of hanging:

$ go run .
failed after 100ms: Get "http://127.0.0.1:62035": context deadline exceeded
is DeadlineExceeded: true

The failure is clean and typed — errors.Is(err, context.DeadlineExceeded) is true — so you can distinguish a timeout from any other error and react to it. The other half of this rule is the server you expose. The zero value of http.Server has no timeouts at all, which means a slow client can open a connection and hold it open indefinitely. Always set them:

srv := &http.Server{
	ReadHeaderTimeout: 5 * time.Second,
	ReadTimeout:       10 * time.Second,
	WriteTimeout:      10 * time.Second,
	IdleTimeout:       60 * time.Second,
}

Retries with backoff and jitter

A transient failure — a dropped connection, a brief 503 — often succeeds on the next try. So you retry. But a naive retry loop is a loaded gun: the moment a popular dependency stumbles, every client retries at once, and the synchronized stampede keeps it down. The fix is two-part. Exponential backoff spaces the attempts out, doubling the wait each time. Jitter randomizes each wait so a thousand clients don’t all retry on the same tick.

// Exponential: base * 2^(attempt-1), capped.
backoff := base * time.Duration(1<<(attempt-1))
if backoff > cap {
	backoff = cap
}
// Full jitter: sleep a random duration in [0, backoff).
sleep := time.Duration(rng.Int63n(int64(backoff)))

Run against a stub that fails three times and then recovers, with a 100ms base capped at 2s:

$ go run .
attempt 1: 503 service unavailable — backing off 31ms (of max 100ms)
attempt 2: 503 service unavailable — backing off 144ms (of max 200ms)
attempt 3: 503 service unavailable — backing off 102ms (of max 400ms)
attempt 4: ok
result: <nil>

Watch the cap climb — 100ms, 200ms, 400ms — while the actual sleep is a random draw beneath it. That randomness is the whole point: it smears a fleet of retriers across the window instead of bunching them. Two guardrails matter as much as the math. Cap the backoff, or the wait grows unbounded. And only retry idempotent operations — a GET or a PUT you can safely repeat, never a POST that charges a card twice. If you can’t guarantee the call is safe to repeat, you can’t safely retry it.

Rate limiting

Backoff protects a dependency from your retries; a rate limiter protects it — and you — from your normal traffic. The standard tool is golang.org/x/time/rate, a token-bucket limiter you add with go get golang.org/x/time/rate. You configure it with a refill rate and a burst size, and it answers one question: may this request proceed right now?

There are two ways to ask. Allow is non-blocking: it returns true if a token is free and false if not, so you can drop or reject the request immediately. Wait blocks until a token frees up, pacing a caller instead of dropping it. Here is a limiter set to 5 events per second with a burst of 3, asked both ways:

lim := rate.NewLimiter(rate.Limit(5), 3)
for i := 1; i <= 6; i++ {
	if lim.Allow() { /* proceed */ } else { /* throttle */ }
}
$ go run .
== Allow(): non-blocking, drop when empty ==
req 1: allowed
req 2: allowed
req 3: allowed
req 4: THROTTLED
req 5: THROTTLED
req 6: THROTTLED
== Wait(): blocks and paces ==
req 1: ran at +0s
req 2: ran at +0s
req 3: ran at +0s
req 4: ran at +200ms
req 5: ran at +400ms

The burst bucket starts full, so the first three requests pass instantly either way. After that Allow drops them, while Wait paces them out at one every 200ms — exactly the 5-per-second refill. Use Allow on an inbound edge where shedding load is correct, and Wait on an outbound edge where you want to respect a downstream’s limit without failing your own request.

A circuit breaker

Retries and rate limits assume a dependency that mostly works. A circuit breaker is for the one that has stopped working. When a service is down, continuing to call it — even with backoff — wastes your time, ties up goroutines, and delays every request that has to time out before failing. A breaker notices the sustained failure and fails fast instead: it stops calling for a while, gives the dependency room to recover, then cautiously probes.

It has three states. Closed is normal; calls pass through and failures are counted. After a threshold of consecutive failures it trips to open, where calls are rejected instantly without touching the dependency. After a cooldown it moves to half-open and allows a single probe: if the probe succeeds it closes, and if it fails it opens again.

$ go run .
t+0s  call         -> state=closed    err=upstream down
t+100ms call         -> state=closed    err=upstream down
t+200ms call         -> state=open      err=upstream down
t+300ms call(fast-fail) -> state=open      err=circuit breaker is open
t+1s  call(fast-fail) -> state=open      err=circuit breaker is open
t+2.5s probe        -> state=closed    err=<nil>
t+2.6s call         -> state=closed    err=<nil>

Three failures trip it. While open, the next two calls return circuit breaker is open without calling the upstream at all — that is the fast-fail, and it is what protects you. After the 2-second cooldown the breaker allows one probe; the dependency has recovered, the probe succeeds, and the breaker closes and resumes normal traffic. The dependency got the breathing room it needed, and your service spent no time waiting on calls that were doomed to fail.

singleflight: collapse duplicate work

A different failure mode isn’t a dependency falling over — it’s a dependency getting hammered by your own service asking it the same thing many times at once. A hot cache key expires, and in the same instant a hundred in-flight requests all miss, all turn around, and all fire the identical expensive query at the database. That’s a cache stampede (or thundering herd), and the database that was fine a second ago now takes a hundred copies of one query. golang.org/x/sync/singleflight is the fix: it ensures that for a given key, only one execution of the work runs at a time, and every concurrent caller shares that single result.

The API is one method. g.Do(key, fn) runs fn if no call with that key is in flight, or waits and returns the in-flight call’s result if one is. It hands back the value, an error, and a shared bool telling you whether the result was shared with other callers:

var g singleflight.Group
v, err, shared := g.Do("cache-key", func() (any, error) {
	return fetchFromDB() // runs once even under a stampede
})

To prove it collapses, launch ten goroutines that all call Do on the same key against a function that counts its own invocations and sleeps long enough for the pile-up to form:

$ go run .
goroutines:        10
load() ran:        1 time(s)
callers w/ shared: 10 of 10
every result:      "the-answer"

Ten goroutines asked; the work ran once; all ten got the same answer, and every one of them saw shared == true. The database saw a single query instead of ten. Two sharp edges keep it honest. Do blocks the caller until the shared call returns, so one slow execution stalls everyone waiting on that key — reach for DoChan with a select on the request’s context when you need a per-caller timeout. And because the result is shared, a transient error is shared too: all ten callers get the same failure, so you can’t retry them independently. singleflight deduplicates work; it does not make that work more reliable, which is what the rest of this chapter is for.

Panic recovery, one last time

There is one failure that comes from inside: a panic in a request handler. Left alone, it unwinds the goroutine serving that request and drops the connection with no response. The recovery middleware from the Building series is the guard, and it belongs on every server you run: a deferred recover turns the panic into a logged 500 and lets the process keep serving.

func recoverMW(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		defer func() {
			if err := recover(); err != nil {
				log.Printf("recovered panic on %s %s: %v", r.Method, r.URL.Path, err)
				http.Error(w, "internal server error", http.StatusInternalServerError)
			}
		}()
		next.ServeHTTP(w, r)
	})
}

Drive a panicking route and then a healthy one against the same server:

$ go run .
2026/07/31 16:39:06 recovered panic on GET /boom: handler exploded
GET /boom  -> 500 "internal server error\n"
GET /ok    -> 200 "still alive\n"

/boom panics and comes back a clean 500; /ok answers 200 on the very next request. One bad handler did not take the server down with it. That is the entire job of the pattern, and it is why it is not optional.

Final thoughts: the whole track, closed

Four series ago this track opened with package main and a line that printed “Hello, Go.” It is worth looking back at the distance covered, because it forms one arc rather than four topics.

The Ground Up series was the language itself: types and zero values, structs and methods, interfaces satisfied implicitly, errors as ordinary values you check rather than exceptions you catch, and the toolbox — build, test, fmt, vet — that ships in the box. You learned to write Go.

Concurrency was the engine underneath: goroutines that cost almost nothing, channels that pass data and, more deeply, establish the happens-before edges that make concurrent memory access defined rather than merely lucky. select, the sync toolkit, context, and the race detector that enforces the memory model instead of trusting your intuition. You learned to make Go run in parallel without lying to yourself about it.

Building Real Things turned the language on the standard library: HTTP servers and clients, JSON in and out, files, testing done properly, and a small REST service assembled from nothing but net/http, encoding/json, sync, and embed — no framework required, because the library is broad and coherent enough not to need one. You learned to build something that works.

And Production — this series — was the gap between works on my machine and survives real traffic: project layout, structured logging with slog, configuration pulled out of the code, graceful shutdown, pprof when something is slow or leaking, observability, build and release, containers, the runtime knobs from the last chapter, and the reliability patterns in this one. You learned to run it.

That is the arc: from a single printed line to a service that is structured, observable, tuned, and resilient — one that logs what it does, shuts down without dropping requests, tells you when it’s slow, and stays standing when its dependencies fall over. Go gave you a small language on purpose, so that the interesting difficulty would live in your problem and not in the syntax. You have the whole of it now: the language, its concurrency model, its library, and the discipline to run it in production. There is nothing left to teach and everything left to build. Go write something real.

Comments