A Proxy in Every Pod Is a Restart in Every Pod

Suppose you decide to put the bookshop into a classic Istio mesh. You label the namespace, you apply, and then you do the thing the docs tell you to do next:

$ kubectl rollout restart deployment -n bookshop

Read that command again, because it is the entire argument of this chapter compressed into one line. To adopt the mesh, you restarted every pod you own. Not upgraded them, not reconfigured them — bounced them, through a rolling update, with everything a rolling update entails. The dropped requests we spent half a chapter measuring. The preStop hooks that had better be there. The database connections that get re-established. The JVMs that need to warm up again. The three services whose readiness probes are subtly wrong and that you are about to find out about.

And when a CVE lands in the proxy — which it will, because it is a large C++ network program facing hostile traffic — you will run that command again. And when you upgrade the mesh from 1.30 to 1.31, which you must do within about six months, you will run it again.

That is not a criticism of the sidecar model so much as a description of it. The sidecar model works, it is mature, it is what nearly every Istio tutorial on the internet teaches, and for a decade it was the only way anybody knew how to build a mesh. But the restart is not incidental to it. The restart is structural, and understanding exactly why is the fastest way to understand why Istio built a second data plane.

Why the sidecar has to restart the pod

A sidecar mesh works by putting an Envoy proxy inside every one of your pods — next to your container, sharing its network namespace. Because they share a network namespace they share a loopback interface and a set of iptables rules, so the proxy can capture everything entering and leaving the pod. Your application, which opened a perfectly ordinary socket to http://catalog, never knows.

The proxy gets in there by injection. You label a namespace istio-injection=enabled, and a mutating admission webhook — a thing you met in the chapter on extending Kubernetes, where custom resources and controllers turned out to be all that Istio is — intercepts every pod creation request on its way through the API server and rewrites the pod spec, adding a container you did not write.

That is a genuinely elegant mechanism — and it has a consequence that follows with the force of arithmetic: a pod’s spec is fixed at creation. You cannot add a container to a running pod. The webhook only ever sees pods being created, so the only pods that get a proxy are the pods created after you turn injection on. Which means every pod that already exists has to be destroyed and replaced. There is no third option.

This is the same law we ran into in the Kubernetes book, and it is worth restating in its general form because it explains half the surprises in this business: a change to the pod template applies to the pods the change creates, never to the pods the change destroys. Turning on injection is a pod-template change wearing a namespace label as a disguise. Turning it off is too — removing the mesh from a namespace also requires bouncing every pod in it, so the escape hatch has the same cost as the entrance.

Everything else uncomfortable about sidecars descends from that one fact.

The rest of the sidecar bill

Two processes, one lifecycle, no coordination. A pod’s containers start in parallel and Kubernetes has, historically, made no promises about the order. So your application container can come up, immediately try to call http://catalog, and fail — because the Envoy next to it has not finished getting its configuration from the control plane, and the iptables rules are already redirecting the call into a proxy that does not yet know where catalog is. Istio’s answer is a setting called holdApplicationUntilProxyStarts, which does precisely what the name says, at the cost of adding the proxy’s startup time to every pod’s startup time. (I did not run a sidecar mesh in this lab, so take the flag name and the failure shape as documented behaviour rather than something I reproduced.)

The Job that never finished. Past tense, and the tense is the point.

Run a Job with an injected sidecar, on an older Istio. Your container does its work and exits zero. The Envoy does not exit — because Envoy is a server, and servers do not exit — so the pod never reaches Completed, the Job never finishes, and your CronJob quietly piles up Active pods until somebody notices a week later. Generations of engineers bodged a curl -X POST localhost:15020/quitquitquit onto the end of their entrypoint script to work around it. It is the most-cited grievance against sidecars, and you will still hear it in architecture reviews.

It is fixed. Kubernetes grew native sidecar containers — an init container with restartPolicy: Always, which the kubelet starts before the app containers and terminates after them (the pods chapter showed the mechanism) — and Istio 1.30 injects the proxy that way by default. In the sidecar chapter of the next book I run a Job with a sidecar and watch it come back Complete, 1/1, in seven seconds.

I am telling you this in the chapter that is supposed to be selling you ambient, because a book that repeats a fixed bug as a current one is worth nothing. The startup-ordering hole and the Job hole are both closed by the platform. If you are going to prefer ambient — and this book does — prefer it for the reasons that are still true: the restart it takes to adopt a sidecar mesh, and the cost, which the next section gets to and which the cost chapter measures at forty megabytes of Envoy per pod against a two-megabyte app.

You pay for L7 whether or not you use L7. This is the one I find most persuasive. Envoy is a full Layer 7 proxy — it parses HTTP, it holds routing tables, it can match on headers and rewrite paths and retry idempotent methods. Now look at the bookshop. postgres speaks the Postgres wire protocol, which is not HTTP; there is nothing for an HTTP proxy to do with it beyond pass bytes through. And yet in a sidecar mesh, postgres gets an Envoy — the same Envoy, with the same memory, holding the same routing config for every service in the mesh, because by default every proxy is told about every destination in case it needs to talk to one.

That last clause is the real scaling problem — and it is why a mature sidecar deployment always ends up with Sidecar resources or discoverySelectors trimming what each proxy is allowed to know about. I did not measure per-pod proxy memory in this lab, so I will not quote you a number; the shape of the cost is the point. It is per-pod, it is proportional to the size of the mesh’s config rather than to anything your pod actually does, and it is charged to every workload you own — including the ones that will never make an HTTP request in their lives.

And the blast radius is everything. There is an Envoy in the path of every request into and out of every pod. A bad config push, a proxy bug, a certificate rotation that goes wrong — the failure is in all of your pods, simultaneously, and the remedy is very often another fleet-wide restart.

To be fair to it, the sidecar model buys real things with that bill. Every pod gets the full Envoy feature set, which means every pod can do L7 routing, header matching, retries, and fine-grained authorization without any further deployment. The blast radius is per-pod, which cuts the other way too — one broken proxy takes down one pod, not one node’s worth of pods. It works with any CNI and makes few assumptions about the node. And, decisively for the next several years, it is the model every existing document, dashboard, blog post, and colleague’s mental picture assumes. That last one is not a technical advantage, and it is nonetheless the strongest argument for sidecars there is.

Ambient: split the job in two

Ambient mode starts from an observation that seems obvious once somebody has said it out loud.

The two things a mesh does are not the same kind of thing. Encrypting and authenticating a connection is a Layer 4 job — it works on TCP, it does not need to parse anything, it is the same for HTTP and Postgres and Redis and gRPC, and everybody wants it. Routing on a header, retrying an idempotent request, denying a DELETE — those are Layer 7 jobs, they require a full HTTP proxy, and only some of your services need them.

The sidecar model bundles both into one process and hands that process to every pod. Ambient unbundles them — and once you have seen the split, the old design stops looking inevitable.

ztunnel is a per-node DaemonSet — one pod on every node, exactly like kube-proxy and your CNI, and for the same reason: it programs and serves the local machine’s traffic. It does Layer 4 and only Layer 4. It terminates and originates mutual TLS, it presents and verifies the workload’s SPIFFE identity, and it can enforce authorization at the level of “may this identity open a TCP connection to that workload’s port”. It is not Envoy; it is a purpose-built proxy written in Rust, which is a project design fact rather than something my cluster told me, and the reason it matters is that it is deliberately small — it holds no HTTP routing tables because it does not do HTTP.

Every pod in an enrolled namespace gets ztunnel’s protection. Automatically. With no sidecar, no injected container, and — this is the part that changes everything — no restart.

waypoint is the other half. It is an Envoy — deployed as an ordinary Deployment, running as an ordinary pod — that you opt into for a namespace or for an individual service. It does the Layer 7 work: HTTP methods and paths and headers, retries, timeouts, request-level traffic splits, L7 authorization. You deploy one where you need one. If postgres never needs L7 — and it does not — then postgres never gets a waypoint, and no Envoy exists anywhere in its path. Not a small one. Not an idle one. None.

The consequence is the thesis of this entire book, so I am going to state it as flatly as I can:

In ambient, mutual TLS and workload identity cost you nothing per pod and require no restarts. You pay for Layer 7 only where you use it.

That is not a marketing line. It is a description of the deployment topology. There is no Envoy in your pods, so there is nothing to inject; there is nothing to inject, so there is nothing to restart; there is no per-pod proxy, so there is no per-pod memory; there is an Envoy only where you asked for one.

What that actually looked like

The claim is checkable, so I checked it. istioctl install --set profile=ambient puts exactly three things in the cluster — istiod, a DaemonSet called istio-cni-node, and a DaemonSet called ztunnel, both running one pod per node on a three-node cluster. No sidecar injector. No Envoy anywhere. The next chapter takes that install apart properly.

Then enrolling the bookshop — six running pods, mid-traffic — was one command:

$ kubectl label namespace bookshop istio.io/dataplane-mode=ambient

Afterwards, every pod had the same name, the same startTime, zero restarts, and still exactly one container. The shop kept answering 200s throughout, and istioctl reported the transport protocol between every pair of workloads as HBONE — an HTTP/2 CONNECT tunnel over mutual TLS, which is the encrypted overlay that ztunnel builds between nodes. Six pods went from plaintext and anonymous to encrypted and cryptographically identified, and not one of them was restarted, rescheduled, or even aware of it.

Hold the two commands next to each other, because this is the whole comparison:

# sidecar
$ kubectl label namespace bookshop istio-injection=enabled
$ kubectl rollout restart deployment -n bookshop     # <- every pod dies

# ambient
$ kubectl label namespace bookshop istio.io/dataplane-mode=ambient

There is no second command. That is not a convenience — it is a category difference in what adopting a mesh is. In the sidecar model, enrolling your fleet is a deploy: a change window, a rollback plan, and a conversation with whoever owns the noisiest service. In ambient it is a label — and if you do not like the result, you remove the label.

Where the restart went

Something has to redirect the pod’s traffic to ztunnel, and that something has to be privileged. In the sidecar world, the classic answer was an init container — istio-init — that ran in the pod with NET_ADMIN and NET_RAW and wrote iptables rules into the pod’s own network namespace. That is a privileged container in every pod you run, which security teams reliably dislike, and it is another thing that only happens at pod creation.

In ambient, that job belongs to istio-cni-node, the third component. It is a DaemonSet, so it runs once per node, as a node-level agent with the privileges it needs — and it programs the node’s networking so that traffic to and from enrolled pods is redirected into that node’s ztunnel. It can do this to a pod that is already running, because it is operating on the node’s network configuration, not on the pod’s spec. Nothing about the pod object changes. Nothing needs to be re-created.

That is where the restart went. It did not get optimized away; the privileged, traffic-capturing work got moved out of the pod and up to the node, where it can be done once, for everybody, at any time.

Which brings us to the honest part.

What ambient is worse at

I would not trust this book if it stopped there, so here is the other column of the ledger.

The failure domain moved; it did not vanish. In the sidecar model a broken proxy breaks one pod. In ambient, ztunnel is a single process handling Layer 4 for every enrolled pod on its node. If ztunnel on k8s-lab-worker is unhealthy, every meshed pod scheduled there has a problem at once. This is exactly the same bargain you already accepted with kube-proxy and your CNI — both of which are per-node and both of which take the node’s pods down with them when they fail — so it is a familiar bargain rather than a new one. But it is a bargain, and anyone who tells you ambient has no blast radius is selling something.

The waypoint is a real hop and a real thing to operate. When you do need L7, the traffic goes through an Envoy that lives in a different pod, quite possibly on a different node from both the client and the server. That is a network hop you did not have with a sidecar, where the proxy was in the pod. It is also a Deployment: it needs replicas, resources, a scaling story, and somebody to notice when it is unhealthy. Ambient does not make Envoy go away — it makes Envoy optional and shared instead of mandatory and per-pod, and shared things need capacity planning.

Layer 7 in ambient is newer. Sidecar Istio has had a decade of people running weird protocols and awkward retry policies through it in production. The waypoint path has not. Some features still assume sidecars, some ecosystem tooling assumes sidecars, and — a real one you will hit in this book — Gateway API’s standard channel does not yet have every knob you want, so a couple of things still require the classic VirtualService and DestinationRule resources. Series 3 of this track is about the sidecar world you will inherit, and about migrating between the two, and it exists because that world is not going anywhere for years.

And most of the documentation you will find is about the other model. Which is not a property of ambient at all. It is a property of you, reading the internet, in 2026 — and it is the single most likely thing to hurt you.

The confusion that will get you

Here is the specific way that hurts, and it is the reason chapter 11 of this book exists in the shape it does. I am putting it here, in chapter 2, because if you skim one paragraph in this series it should be this one.

You write an authorization policy that says: orders may GET /books on the catalog, and nothing else. You apply it. You test it by sending a DELETE /books from orders, and you wait for the 403 that every tutorial you have ever read shows you.

In an ambient mesh with no waypoint on catalog, that DELETE returns 200.

Not 403. Not an error. Not a warning at apply time. 200 — the request went through, the delete happened, and your policy enforced nothing at all. The reason is now, I hope, structural rather than mysterious: DELETE is an HTTP method, HTTP methods are Layer 7, ztunnel does Layer 4, and there is no waypoint in the path — so there is nothing in the request’s path capable of reading the method your rule is written against. An L7 rule with no waypoint is not a stricter policy. It is an unenforced one, and it fails open.

Deploy a waypoint for that service — one command, nothing else changed, same policy, same caller — and the same DELETE comes back 403.

And the other half of the same confusion, which is just as disorienting in the other direction. Write a Layer 4 policy instead — this identity may connect to that workload, full stop — and call it from a pod that is not allowed to. You do not get a 403. You get this:

http=000, curl exit=56

000 is an old friend: it is what curl prints when there was no HTTP response at all. Exit 56 is “connection reset by peer”. ztunnel does not send you a status code, because sending a status code would mean speaking HTTP, and ztunnel is not speaking HTTP — it is a Layer 4 proxy, and the Layer 4 way to say no is to drop the connection on the floor.

So: an L4 denial is a connection reset. An L7 denial is an HTTP 403. Every sidecar tutorial shows a 403 for everything, because in a sidecar mesh there is a full Envoy in the path of every request and it is always able to answer in HTTP. A reader who brings that expectation to an ambient mesh will spend an afternoon waiting for a 403 that is never coming — or, far worse, will see a 200, conclude the policy is broken and the request should have been allowed, and move on with an unenforced rule in production.

Both of those results are run-verified in the lab this book was written on, and chapter 11 does nothing but take them apart. I want you to have seen them before you write your first policy.

Choosing, honestly

The comparison, laid out:

	sidecar	ambient
proxy placement	Envoy in every pod	ztunnel per node (L4) + optional waypoint (L7)
enrolling a workload	pod restart, always	a namespace label, no restart
removing the mesh	pod restart, always	remove the label
cost per pod	one full Envoy, sized by mesh config	none
L7 features	everywhere, for everyone	only where you deploy a waypoint
upgrading the data plane	restart the fleet	roll a DaemonSet
blast radius	one pod per proxy	one node per ztunnel; a shared waypoint
an L4 denial looks like	HTTP 403	connection reset (curl exit 56)
maturity	a decade	newer, and moving fast
what the internet assumes	this one	not this one

If you are starting from nothing today, on a supported Istio, I think ambient is the right default, and this book is written as though you agree. The adoption cost is genuinely close to zero, the steady-state cost is genuinely lower, and the direction of the project is not ambiguous.

Sidecars remain the right answer in a few real cases. You already have a sidecar mesh and it works — do not rip it out because of a blog post; migration is a project, and Series 3 is about doing it deliberately. You need per-pod L7 granularity that a service-scoped waypoint cannot express. You depend on a feature or an ecosystem tool that has not made the crossing yet. Or your organization’s operational knowledge, dashboards, and runbooks are all sidecar-shaped, and the cost of retraining everybody exceeds the cost of the restarts — which is a legitimate engineering answer, and one I would rather you gave out loud than pretended was a technical one.

Final thoughts

The thing I find genuinely instructive about ambient is not that it is newer or lighter. It is where it chose to put the boundary.

The sidecar model made one decision — put a full proxy next to every workload — and everything else followed: the restarts, the memory, the lifecycle races, the fleet-wide upgrades, the privileged init container. It is an entirely coherent design, and for years it was the best one available. Ambient made a different single decision: separate the part everybody needs from the part only some people need, and put them in different places. Everything else follows from that: the node agent, the zero-restart enrollment, the opt-in Envoy, the free mTLS — and, unavoidably, the two different flavours of “no” that will confuse you in chapter 11.

That is a good lesson to carry beyond service meshes, because it is the lesson of the pod and the lesson of the Deployment and the ReplicaSet too. Most of what feels arbitrary in a piece of infrastructure is the downstream consequence of one early decision about where a boundary goes, and when a system finally becomes easy to reason about, it is usually because somebody moved a boundary rather than because they added a feature.

Enough theory. Let us install the thing — and let us talk about the six-month clock that starts the moment we do.

Next: The Mesh That Changed Nothing