Draining the Server: Shutdown That Doesn't Drop Requests

A process does not get to choose when it dies, but it does get a warning. When an orchestrator decides to stop your service — a deploy, a scale-down, a node drain — it sends SIGTERM and waits a few seconds before it sends SIGKILL, which cannot be caught. Those few seconds are the whole game. Use them to stop accepting new connections and let the requests already in flight run to completion, and nobody notices the restart. Ignore them, and every request being served at that instant dies with a dropped connection, a truncated response, or a client-side timeout. This chapter is about spending that grace period well. Everything below was compiled and run against Go 1.26.5.

The default is abrupt

Left to itself, a Go HTTP server does not shut down gracefully; it just stops existing. http.ListenAndServe blocks forever, and when the process receives an uncaught SIGTERM the runtime terminates immediately. Any goroutine currently inside a handler is gone mid-sentence, its response half-written, its database transaction neither committed nor rolled back. The fix has three moving parts, and they have to work together: notice the signal, stop ListenAndServe, and give the active handlers a bounded window to finish.

Turning a signal into a context

The cleanest way to notice the signal is signal.NotifyContext, which hands you a context.Context that cancels the moment one of the named signals arrives:

ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()

os.Interrupt is Ctrl-C during local development; syscall.SIGTERM is what Kubernetes, systemd, and Docker actually send. The returned ctx starts alive and flips to done on the first matching signal, so <-ctx.Done() is your “time to shut down” trigger. The stop function detaches the handlers again; calling it after the signal arrives restores the default behavior, which matters for a reason we will get to.

Run the server off the main goroutine

ListenAndServe blocks, so if you call it on the main goroutine you have nowhere left to wait for the signal. Push it into its own goroutine and keep the main line free:

srv := &http.Server{Addr: ":8080", Handler: mux}

go func() {
	if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
		fmt.Println("server: unexpected error:", err)
	}
}()

<-ctx.Done() // block here until a signal fires

The errors.Is(err, http.ErrServerClosed) check is not optional bookkeeping; it is the contract. When you later call Shutdown, ListenAndServe returns a specific sentinel error, http.ErrServerClosed, to tell you the stop was deliberate. Any other error means the listener failed for a real reason (a port already in use, for instance). Confirmed directly:

ListenAndServe returned: http: Server closed
errors.Is(err, http.ErrServerClosed): true

So the idiom is: treat ErrServerClosed as success and everything else as a genuine failure. Skip the check and you either log a scary “error” on every clean shutdown or, worse, swallow a real bind failure.

Shutdown drains, it does not chop

Once the signal fires, call srv.Shutdown with a context that carries a deadline:

<-ctx.Done()
stop() // restore default handling; a second Ctrl-C now kills instantly

shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()

if err := srv.Shutdown(shutdownCtx); err != nil {
	fmt.Println("shutdown: forced:", err)
}

Shutdown does two things in order. First it closes all the open listeners, so no new connection is accepted — the server is now invisible to any client that hasn’t already connected. Then it waits for every in-flight request to return, blocking until the last handler finishes or the context’s deadline hits, whichever comes first. If the deadline wins, Shutdown returns a non-nil error (context deadline exceeded) and abandons whatever is still running. That bound is deliberate: a stuck handler must not hold the shutdown hostage past the orchestrator’s own patience, because after its grace period the orchestrator sends SIGKILL regardless. Size the timeout to fit comfortably inside that window — Kubernetes defaults to 30 seconds, so 10 or 15 is a safe drain budget.

The stop() call right after ctx.Done() restores the default signal disposition. Now a second SIGTERM (an impatient operator hitting Ctrl-C twice) kills the process immediately instead of being swallowed by our handler. You asked for a graceful shutdown once; the second signal means “I’ve changed my mind, die now,” and honoring it is the polite thing to do.

Watching it actually work

The claim is that a request in flight survives the signal. Here it is demonstrated end to end: a handler that takes three seconds, a client that fires it, and a SIGTERM delivered to our own process 300 ms after the request lands — well before the handler is done.

mux.HandleFunc("GET /slow", func(w http.ResponseWriter, r *http.Request) {
	fmt.Println("slow handler: started, working for 3s")
	select {
	case <-time.After(3 * time.Second):
		fmt.Fprintln(w, "done")
		fmt.Println("slow handler: finished, wrote response")
	case <-r.Context().Done():
		fmt.Println("slow handler: request context cancelled")
	}
})

// elsewhere, a goroutine delivers the signal mid-request:
go func() {
	time.Sleep(600 * time.Millisecond)
	p, _ := os.FindProcess(os.Getpid())
	_ = p.Signal(syscall.SIGTERM)
}()

Sending the signal to os.Getpid() is a trick to keep the demo self-contained; in production the signal comes from outside and you never write that goroutine. The output tells the whole story:

server: listening on :8080
client: firing GET /slow
slow handler: started, working for 3s
signal: delivering SIGTERM to self
shutdown: signal received, draining
server: ListenAndServe returned
slow handler: finished, wrote response
client: got 200 OK after 3.004s

Read the order carefully, because it is the entire point. The signal arrives and shutdown begins while the handler is still working. ListenAndServe returns right away — the listener is closed, no new request will ever be accepted. But the handler already running keeps going, finishes its full three seconds, and writes its response. The client gets a clean 200 OK after 3.004 seconds. Only then, with the last request drained, does the process exit. The signal did not interrupt the work; it started a countdown that the work comfortably beat.

The handler’s own context is the other half

Notice the handler also selects on r.Context().Done(). A request’s context is cancelled when the client disconnects, and Shutdown does not cancel it — draining means letting requests finish, not tearing them down. But wiring your slow operations to r.Context() matters for a different case: if the shutdown deadline expires and the process is about to be killed anyway, a handler that respects its context can bail out of a pointless computation instead of doing three more seconds of work nobody will read. Graceful shutdown drains what can finish in time; context cancellation is how the rest stops wasting effort.

Sharp edges

A few things bite people once this is wired up.

• Long-lived connections don’t drain. Shutdown waits on active requests, but a hijacked connection or an idle keep-alive WebSocket has no request in flight for it to wait on. WebSockets in particular need their own close signal; Shutdown will hit its deadline waiting for them otherwise.
• Background work is not covered. Shutdown knows about HTTP handlers, nothing else. A goroutine you spawned to flush a buffer or drain a queue has to be tracked and waited on separately — typically with a sync.WaitGroup you Wait on after Shutdown returns.
• Close is the guillotine, not Shutdown. srv.Close() also exists and it does the opposite thing: it slams every connection shut immediately, in-flight requests and all. It is for “get out now,” not “get out cleanly.” Reach for Shutdown unless you specifically want the abrupt version.
• The readiness probe should flip first. In Kubernetes there is a race between “the pod stops passing readiness” and “the load balancer stops routing to it.” A robust shutdown fails its readiness check, sleeps a beat to let the endpoints update, and only then calls Shutdown — otherwise new requests can still arrive during the drain.

Final thoughts

Graceful shutdown is four lines of ceremony around one real idea: a SIGTERM is a request, not a command, and you have a few seconds to answer it well. signal.NotifyContext turns the signal into a context, ListenAndServe runs off the main goroutine so the main line can wait, and Shutdown with a bounded timeout closes the door and then drains the room. Treat http.ErrServerClosed as the success signal it is, keep the drain budget inside the orchestrator’s grace period, and remember that anything that isn’t an HTTP handler — background goroutines, WebSockets, queues — you have to drain yourself. Get it right and a deploy is invisible to your users. Get it wrong and every rollout sheds a handful of failed requests that no dashboard will ever quite explain.

Next: find the hot spot, do not guess it — profiling with pprof, and reading a real CPU profile instead of guessing where the time goes.