Are You Alive, or Just Breathing? Health Checks That Tell the Truth

An orchestrator cannot see inside your process. It knows the pod is running because there is a PID, but a PID tells it nothing about whether your service can actually answer a request. Maybe the process is up but still loading a 400 MB model. Maybe it’s up and serving fine. Maybe it’s up but wedged in a deadlock, holding a connection it will never respond on. To the kernel all three look identical: alive. The only way the platform learns the difference is if your code tells it, and the channel for that is a health check — an HTTP endpoint the orchestrator polls to ask a specific question and act on the answer.

The trap is thinking there is one question. There are three, they have different answers, and wiring them together is one of the most common ways a healthy service still sheds traffic. This chapter untangles them and then builds the endpoints, compiled and run against Go 1.26.5.

Three probes, three different verbs

Kubernetes formalized the distinction, but the idea applies to any platform that supervises a process. There are three kinds of probe, and the reason they exist separately is that each one triggers a different action on failure.

Liveness asks: is this process fundamentally broken? A failing liveness probe means restart me. The orchestrator kills the container and starts a new one. This is the nuclear option, reserved for states you cannot recover from in place — a deadlock, a corrupted internal state, a goroutine leak that has eaten all your memory. The critical rule follows directly from the consequence: a liveness check must be cheap and independent. It must not touch the database, call an upstream, or check anything outside the process. If it does, a brief database outage fails every pod’s liveness probe at once, and the orchestrator responds by restarting your entire fleet — turning a recoverable dependency blip into a full outage of your own making. Liveness answers one thing only: is the process itself still capable of serving? While the process is alive and its request loop is turning, it should return 200.

Readiness asks: should I receive traffic right now? A failing readiness probe means take me out of the load balancer, but leave me running. No restart. The pod stays alive; the platform simply stops routing new requests to it until it recovers. This is the probe that should check dependencies — if you can’t reach the database you depend on, you can’t serve a useful response, so you should bow out of rotation until you can. Readiness is temporary and reversible by design: a pod flips out of rotation when a dependency is down and flips back in when it recovers, all without dying.

Startup asks: has this thing finished booting yet? It exists to solve a timing conflict. A service that takes thirty seconds to warm up would fail an aggressive liveness probe during those thirty seconds and get killed before it ever came up, restarting forever. The startup probe holds the other two off: while it is failing, the liveness and readiness probes are not even checked. Once it passes once, the platform switches to the normal liveness and readiness cadence and never consults the startup probe again. It is a one-time gate that says “don’t judge me until I’ve booted.”

The short version: liveness restarts you, readiness reroutes around you, startup waits for you. Conflate any two and you get the wrong action at the worst time.

Two endpoints

In practice, two HTTP handlers cover it. /healthz is liveness — dead simple, always 200 while the process is alive, no dependency checks. /readyz is readiness — it checks whatever you need to serve and returns 503 when you can’t. (Startup is usually the same /readyz endpoint pointed at by a startup probe with a longer timeout, so you rarely write a third handler.) The naming with the trailing z is a Kubernetes convention, chosen to avoid colliding with a real route.

mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, r *http.Request) {
	w.WriteHeader(http.StatusOK) // alive if we can answer at all
	fmt.Fprintln(w, "ok")
})

mux.HandleFunc("GET /readyz", func(w http.ResponseWriter, r *http.Request) {
	if !ready.Load() {
		w.WriteHeader(http.StatusServiceUnavailable) // 503: don't route to me
		fmt.Fprintln(w, "draining")
		return
	}
	w.WriteHeader(http.StatusOK)
	fmt.Fprintln(w, "ready")
})

The ready gate here is an atomic.Bool, because the readiness answer is state that other goroutines flip. A real /readyz would also ping its dependencies inside that handler — db.PingContext(r.Context()) and friends — returning 503 the moment one is unreachable. Keep those checks fast and give them their own short timeout; a readiness probe that blocks for ten seconds is its own kind of outage.

The production move: flip readiness before you drain

Here is the part that ties this chapter to graceful shutdown, and the reason readiness is a variable and not a fixed 200. Recall from the shutdown chapter the race that lurks in every rolling deploy: when your pod gets SIGTERM, there is a lag before the load balancer’s endpoint list updates to stop routing to it. During that lag, new requests keep arriving at a process that has already decided to die. Drain immediately and those requests hit a closing door.

Readiness closes that gap. The move is to flip /readyz to not-ready first, pause long enough for the load balancer to notice and pull you from rotation, and only then start draining in-flight requests. The order is the whole point: fail readiness, wait, then Shutdown. Liveness stays 200 the entire time, because the process is not broken — it is deliberately, gracefully leaving.

<-ctx.Done() // SIGTERM arrived
stop()

ready.Store(false) // 1. tell the load balancer to stop routing here

// 2. (in production) sleep a few seconds so the endpoint list updates,
//    then 3. drain the requests already in flight:
if err := srv.Shutdown(shutdownCtx); err != nil {
	fmt.Println("shutdown:", err)
}

The self-contained demo below skips the sleep for speed, but it exercises the state change that matters: probe both endpoints while healthy, deliver a SIGTERM to our own process, and probe again mid-shutdown.

$ go run .
== healthy: both endpoints up ==
/healthz  -> 200 "ok\n"
/readyz   -> 200 "ready\n"

== SIGTERM received: readiness OFF, then drain ==
/healthz  -> 200 "ok\n"
/readyz   -> 503 "draining\n"
drained and exiting

Read the two blocks against each other. While healthy, both endpoints return 200 — the pod is alive and in rotation. After SIGTERM, /healthz is still 200 because the process is perfectly alive, but /readyz has flipped to 503. That single 503 is the pod telling the load balancer “route elsewhere” without telling the orchestrator “restart me.” The traffic drains away cleanly, the in-flight work finishes, and only then does the process exit. Liveness and readiness gave two different answers to two different questions at the same instant, which is exactly what they are for.

Sharp edges

A few things bite once this is wired up.

• Never check dependencies in liveness. It bears repeating because the failure is so tempting to write and so catastrophic when it fires. A liveness probe that pings the database converts a database hiccup into a fleet-wide restart storm. Dependencies belong in readiness, never liveness.
• A flapping readiness check is worse than a failing one. If /readyz toggles in and out on every marginal dependency latency, the pod pinballs in and out of rotation and every routing table thrashes. Add a little hysteresis or a short failure threshold rather than reacting to a single slow ping.
• Give the probe handlers their own timeouts. These endpoints get polled every few seconds forever. If a readiness check can hang, it will eventually hang during an incident, and a hung probe reads as a failure. Bound it.
• Don’t authenticate the probe endpoints. The kubelet calling /healthz has no credentials to offer. Leave these routes open (they expose nothing sensitive) or the probe itself fails with a 401 and the platform kills a healthy pod.

Final thoughts

Health checks are the interface between your process and the system supervising it, and the whole art is answering the right question. Liveness is “am I broken, restart me” — keep it cheap and dependency-free, because its failure is a restart. Readiness is “should I get traffic now” — check your dependencies here, because its failure is a reroute, not a death. Startup is “have I booted yet” — a one-time gate so a slow warm-up isn’t mistaken for a broken process. The endpoints are a few lines each; the discipline is in not blurring them together. And the payoff move is small and specific: flip /readyz to 503 on SIGTERM before you drain, so the load balancer stops sending work to a pod that has already begun to leave. A deploy where every probe tells the truth is a deploy nobody notices.

Next: assume everything fails — the patterns that keep a service standing when the things it depends on don’t.