Six Pods, Zero Restarts, One Label

Enrolling a namespace into an ambient mesh is a single kubectl label. The pods do not restart, do not gain a container, and do not notice — and the sidecar model structurally cannot do that.

Somewhere right now there is a Confluence page titled “Service Mesh Onboarding — Phase 2”, and on it is a table. One row per namespace, one column for the team that owns it, one column for the change window, one column for the rollback plan. The change window is a Thursday at 22:00. The change is: restart every pod in the namespace.

That page exists because of an implementation detail — not a security requirement, not a design constraint of mutual TLS, not anything anybody sat down and chose. It exists because in the sidecar model the proxy is a container in your pod, containers are declared in the pod spec, a pod spec is immutable after admission, and therefore a pod that does not have a proxy can never acquire one. The only way to give a running workload a sidecar is to destroy the workload and create a different one that has the sidecar in its spec. That is what “enrolling a namespace” means with sidecars, and it is why it takes a quarter, a change board, and a Thursday night.

Here is the ambient version, in full:

$ kubectl label namespace bookshop istio.io/dataplane-mode=ambient
namespace/bookshop labeled

That is not a simplified example. That is the command. The bookshop is now in the mesh — every pod in it has a cryptographic identity and every connection between them is mutually authenticated and encrypted — and this chapter is going to spend most of its length proving that claim rather than asserting it, because the claim is large enough that you should not believe it on my say-so.

The before and the after

The bookshop from the Kubernetes book is running in the bookshop namespace: two web pods, two catalog pods, one orders, one postgres. Before the label, I asked the API server for each pod’s name, its ready state, its restart count, the time its containers started, and the list of containers in it:

BEFORE (name / ready / restarts / startTime / containers):
  catalog-8c6d8db8c-2kd49     true   0     2026-07-14T01:45:54Z   catalog
  catalog-8c6d8db8c-7fp5c     true   0     2026-07-14T01:45:53Z   catalog
  orders-db6c9c86c-blbfb      true   0     2026-07-14T01:45:54Z   orders
  postgres-84bfd89b75-lmppf   true   0     2026-07-14T01:45:58Z   postgres
  web-57b4dcbd6b-92qtr        true   0     2026-07-14T02:20:44Z   web
  web-57b4dcbd6b-gqlrz        true   0     2026-07-14T02:20:44Z   web

Then the label. Then the same query again:

AFTER:
  catalog-8c6d8db8c-2kd49     true   0     2026-07-14T01:45:54Z   catalog
  catalog-8c6d8db8c-7fp5c     true   0     2026-07-14T01:45:53Z   catalog
  orders-db6c9c86c-blbfb      true   0     2026-07-14T01:45:54Z   orders
  postgres-84bfd89b75-lmppf   true   0     2026-07-14T01:45:58Z   postgres
  web-57b4dcbd6b-92qtr        true   0     2026-07-14T02:20:44Z   web
  web-57b4dcbd6b-gqlrz        true   0     2026-07-14T02:20:44Z   web

Read the two blocks against each other, line by line, because every column in them is load-bearing.

Same pod names. catalog-8c6d8db8c-2kd49 is the same object it was a minute ago. It was not deleted and recreated — a recreated pod gets a fresh five-character suffix, and if the pod template had changed at all it would also get a fresh ReplicaSet hash, so the middle segment would move too. Neither moved. No controller made a new pod.

Same startTime. 2026-07-14T01:45:54Z, before and after. This is the timestamp the kubelet records when the pod’s containers begin running; it is the closest thing a pod has to a birthday — and it is not the kind of field that survives a restart. The process serving /books in that container has been running continuously across the enrollment. It did not pause, it did not re-exec, it did not reconnect to Postgres.

Zero restarts. Not “one restart, cleanly handled” — zero. The restart counter is per-container and it is monotonic; the containers have never gone down.

Still one container each. This is the column the whole chapter turns on. catalog has one container in it, called catalog. Not catalog plus istio-proxy. Not two. There is no proxy in that pod. There is no init container that rewrote its iptables. The pod’s spec is byte-for-byte what you applied in the Kubernetes book — and if you kubectl get pod -o yaml it, you will find nothing in there that mentions Istio at all.

And during the enrollment, with the label going on, I curled the shop through its NodePort:

$ curl -o /dev/null -w '%{http_code}\n' localhost:30080
200

I want to be precise about what that does and does not prove, because the Kubernetes book taught you to be suspicious of exactly this kind of claim. It is a single request, not a 500-request loop with a status-code histogram of the sort we used to catch the dropped request during a rolling update. What it proves is that the shop was serving through the change. What the six-row table above proves — and this is the stronger claim — is that there was no window in which it could have dropped anything, because nothing was ever taken away. No pod was terminated. No endpoint was removed from a Service. No kube-proxy anywhere had to be told anything. The entire class of rollout race conditions we spent chapter 3 of the Kubernetes book dissecting simply does not arise, because there is no rollout.

Why the sidecar model cannot do this, ever

It is worth being exact about the mechanism, because “sidecars need a restart” is usually stated as a performance complaint and it is not one. It is a consequence of how Kubernetes admission works.

A sidecar gets into a pod through a mutating admission webhook. You met admission in the Kubernetes book’s RBAC chapter as the third gate — after authentication says who you are and authorization says you may do this, admission gets to look at the object and change it or reject it. Istio’s injector registers a MutatingWebhookConfiguration that matches CREATE operations on pods in labelled namespaces. When the API server is about to persist a new pod, it POSTs the pod to the injector, and the injector returns a JSON patch that adds the istio-proxy container, an init container, some volumes, and a pile of annotations.

Look at the operation it matches: CREATE. Not UPDATE. The webhook fires exactly once in a pod’s life — at the instant it comes into existence — and it can only rewrite the object that is being written. It cannot reach out and touch the two hundred pods that already exist, because those pods are not being created; they are just sitting there, already admitted, already scheduled, already running, with a spec that is — for the fields that matter here — immutable. You cannot kubectl patch a container into a running pod. The API server will reject it.

So the sidecar injector’s contract is precise and it is unavoidable: it applies to pods created after you enable it, and to no others. Which makes the enrollment procedure a rolling restart of every workload in the namespace, which makes it a coordinated change with every team that owns one of them, which makes it a Thursday night, which makes it a Confluence page. Every part of that chain follows from the word CREATE in a webhook configuration.

(There is a real subtlety here that has improved and is worth knowing: modern Kubernetes has native sidecar containers — an init container with restartPolicy: Always — which fixed genuinely painful problems around startup ordering and Jobs that would never complete because the proxy never exited. It is a good change. It does not touch this one. A native sidecar is still a container in the pod spec, and it still arrives at CREATE.)

Ambient does not have this problem because ambient does not have this design. The proxy is not in your pod. There is nothing to inject, so there is no injection, so there is no webhook, so there is no restart.

So where is the proxy?

The pods did not change. The traffic is now encrypted and authenticated. Both of those are true, and the only way to hold them in your head at once is to know exactly what the label actually did.

Three things were installed in the previous chapter — and only three: istiod, the control plane; the ztunnel DaemonSet, one pod per node; and the istio-cni-node DaemonSet, also one pod per node. It is the third one that makes enrollment work, and it has the most misleading name in the whole system.

istio-cni-node is not really a CNI plugin in the sense of replacing your cluster’s networking — the bookshop’s cluster still runs whatever CNI the Kubernetes book gave it, and Istio does not displace it. What istio-cni-node does is watch the API server for pods in namespaces labelled istio.io/dataplane-mode=ambient, and for each one, reach into that pod’s network namespace from the node and install redirection rules in it. It runs as a privileged pod on the node with access to the host’s network namespaces, so it can do this to a pod that is already running — from the outside, without the pod’s cooperation and without its knowledge.

The result is that traffic leaving an enrolled pod is redirected — before it reaches the node’s normal routing — to the ztunnel pod on that same node. Traffic arriving at an enrolled pod comes from the local ztunnel. ztunnel then does the mesh’s work: it establishes a mutually authenticated tunnel to the ztunnel on the destination pod’s node, carries the connection inside it, and hands it over locally at the far end.

Sit with the consequence for a second, because it is genuinely strange the first time. Your application opened a plain TCP connection to catalog:80. It did not negotiate TLS. It does not have a certificate. It was not recompiled, reconfigured, or restarted. And that connection is now travelling between two nodes inside mutual TLS, with a client certificate that names the workload that opened it. The pod does not know it is in a mesh. From inside the container, nothing is different — same code, same socket call, same Host header, same everything. The change happened in the node’s network stack, one layer below the boundary the container can see.

That is the whole trick, and it is why the enrollment is a label rather than a project.

Edge-triggered versus level-triggered, one more time

There is a deeper idea underneath the last two sections, and it is one the Kubernetes book hammered on for fifteen chapters: the difference between reacting to an event and reconciling toward a state.

The sidecar injector is edge-triggered. It is a webhook. It fires on an event — a pod CREATE — it does its work in the microseconds it has while the API server waits, and then it is done forever. Everything that was already true when it fired is invisible to it. Everything that becomes true after it fires is somebody else’s problem. That is the model, and it is why enrollment is a fleet restart: the only way to make the event happen for a pod that already exists is to destroy the pod so that a new one can be created and the edge can be triggered.

istio-cni-node is level-triggered. It is a controller with a watch, and it has exactly the shape of every controller in the Kubernetes book — observe the desired state, observe the actual state, close the gap, repeat forever. Desired state: “every pod in an ambient-labelled namespace has redirection programmed in its network namespace.” Actual state: whatever it finds. The gap: six pods that need programming. It closes the gap. It does not care whether those pods were created a second ago or a month ago, because a reconciliation loop has no concept of “before” — it only has now, and now the label is on the namespace and those pods are in it.

Which is why the same mechanism handles the case you did not think to ask about: a pod created after the enrollment. It is not injected — nothing is injected, ever — it is simply reconciled, the same way, at the same layer, by the same loop. Enrollment before creation and enrollment after creation are not two code paths in ambient. They are the same code path, run at different moments, and a controller does not know what moment it is.

This is not a stylistic preference. Every operational property in this chapter falls out of it. No restarts, retroactive enrollment, a working escape hatch that is also a label, reversibility — all of it is what you get for free when the mechanism is a loop instead of a hook. And it is a useful lens to carry into any platform decision: ask whether the thing that configures your workloads is watching state or catching events, because the answer tells you, before you read a single line of its documentation, whether adopting it will require a maintenance window.

Checking that it worked

kubectl get pods will tell you nothing — that is the point of the chapter, and it is also, briefly, quite disorienting. The pods look exactly as they did. You need to ask the mesh, not Kubernetes:

$ istioctl ztunnel-config workload
NAMESPACE   POD NAME                            ADDRESS      NODE              WAYPOINT PROTOCOL
bookshop    catalog-8c6d8db8c-2kd49             10.244.1.57  k8s-lab-worker    None     HBONE
bookshop    catalog-8c6d8db8c-7fp5c             10.244.2.38  k8s-lab-worker2   None     HBONE
bookshop    orders-db6c9c86c-blbfb              10.244.2.39  k8s-lab-worker2   None     HBONE
bookshop    postgres-84bfd89b75-lmppf           10.244.2.42  k8s-lab-worker2   None     HBONE
bookshop    shelf-controller-6f9877c9bd-csd2h   10.244.2.45  k8s-lab-worker2   None     HBONE
bookshop    web-57b4dcbd6b-92qtr                10.244.1.59  k8s-lab-worker    None     HBONE
bookshop    web-57b4dcbd6b-gqlrz                10.244.2.44  k8s-lab-worker2   None     HBONE

(The command lists every workload the mesh knows about; I have cut the output to the bookshop’s namespace. Seven workloads, not six: shelf-controller is the custom controller you wrote in the Kubernetes book’s CRD chapter, which lives in the same namespace and got enrolled along with everything else, without anyone asking it.)

PROTOCOL: HBONE is the column that means “enrolled”. HBONE is the tunnel — HTTP-Based Overlay Network Environment — and it is what ztunnel speaks to other ztunnels. A workload whose protocol is TCP is being handled as plain traffic; a workload with HBONE is inside the mesh, with an identity, with mTLS. Every bookshop pod says HBONE. The label took.

Ignore the WAYPOINT: None column for now. It is not an error and it does not mean anything is missing, but it is the single most convincing false alarm in ambient mode and it deserves its own explanation, which it gets in chapter 7.

The trap: -n does not mean what you think

Now the mistake you are about to make — which I made — and which produces an error message so confidently wrong-looking that it will send you off to re-read the install chapter.

You have the table above. It is long. You want just the bookshop. Every other kubectl and istioctl command in your muscle memory takes -n to scope to a namespace, so:

$ istioctl ztunnel-config workload -n bookshop
Error: failed retrieving: daemonsets.apps "ztunnel" not found in the "bookshop" namespace

Read the error, not the command. It is not saying “no workloads in bookshop”. It is saying it went looking for a DaemonSet called ztunnel in the bookshop namespace and could not find one — which is correct, because ztunnel lives in istio-system.

The ztunnel-config family of commands does not query the API server for workloads. It queries a running ztunnel pod’s admin interface and dumps that proxy’s view of the world. So its -n flag means the namespace where ztunnel is, not the namespace you want to see. It is a flag about the thing being interrogated, not about the thing being reported. Once you know that, the whole istioctl proxy-config / ztunnel-config family stops surprising you: the target is always a proxy, and the flags select the proxy.

What you actually want is either the unfiltered output piped through grep, or --node to pick which node’s ztunnel you ask:

$ istioctl ztunnel-config workload --node k8s-lab-worker

Every ztunnel has a complete view of the cluster’s workloads — istiod pushes the whole set to all of them — so it does not matter much which one you ask. It will matter enormously in the next chapter, where the thing you are asking for is a certificate, and a certificate is not the same on every node.

The namespace is the unit, and the pod is the exception

istio.io/dataplane-mode=ambient on the namespace is the normal way to enrol, and it should be. Namespaces are how Kubernetes draws the line around a team’s blast radius — that was the argument in the namespaces chapter of the Kubernetes book — and “everything this team runs is in the mesh” is a sentence an organisation can actually hold in its head.

The same label works on a pod, and it takes precedence over the namespace’s. That gives you two moves:

  • istio.io/dataplane-mode=ambient on a pod in an unlabelled namespace: pull one workload in, leaving the rest of the namespace alone.
  • istio.io/dataplane-mode=none on a pod in a labelled namespace: push one workload out.

The opt-out is the one you will reach for, and there is one recurring reason. Some workloads are unhappy about having their traffic transparently redirected — the usual suspects are things that already do their own low-level network manipulation, host-networked pods, and anything terminating its own TLS in a way that gets confused by a transparent tunnel. When you meet one, dataplane-mode: none on that pod is your escape hatch — and the fact that it exists as a label rather than a redeployment means the escape hatch is as cheap as the enrollment was.

I labelled a whole namespace and did not exercise the per-pod overrides in the lab, so take the two bullets above as documented behaviour I have not personally run. Everything else in this chapter is output.

What enrollment does not give you

This is the part where you should slow down, because the temptation after a successful kubectl label is to write “mesh: done” on the ticket and move on. Enrollment gave you two things, precisely — identity, and encryption in transit between enrolled workloads. It did not give you authorization. Nothing is being denied. Nothing was ever being denied.

Here is the proof, and it is the baseline for the security chapters. There is a pod called intruder, in a different namespace, in no mesh at all — no label, no ambient, no ztunnel handling its traffic. It curls the bookshop’s catalog service directly:

orders   (in mesh, SA=orders)    -> HTTP 200
intruder (ns outside, no mesh)   -> HTTP 200

Both get in. The mesh happily accepts a plain, unauthenticated, unencrypted connection from a workload it has never heard of. That is not a bug and it is not a misconfiguration — it is Istio’s PERMISSIVE posture, and it is the only sane default, because the alternative is that the moment you type kubectl label, every unenrolled client of every service in that namespace breaks at once. Prometheus scraping your /metrics. A legacy cron job in another namespace. The monitoring sidecar somebody added in 2023 and forgot. A mesh whose enrollment step is “and now everything outside the mesh stops working” is a mesh nobody can adopt incrementally, and incremental adoption is the entire point of making enrollment a label.

So hold both halves:

Enrollment is an authentication event, not an authorization event. Every enrolled pod now has a verifiable identity and every mesh-internal connection is now encrypted. Who may talk to whom has not changed by one byte.

Closing that door is a separate, deliberate act — an AuthorizationPolicy — and it is deliberate on purpose. You get to enrol first, watch the telemetry, discover which callers actually exist (including the three you had forgotten about), and only then start denying. Chapter 6 does the first half of that with an L4 policy, and chapter 11 does the whole thing properly, including the part where the denial does not look like you expect.

Enrollment is also reversible in the obvious way — remove the label, and istio-cni-node un-programs the pods’ redirection. I did not run the un-enrollment, so treat that as the documented behaviour rather than a result, but the shape of it follows from the mechanism: nothing was ever added to the pods, so there is nothing to remove from them.

Final thoughts

The genuinely interesting thing about kubectl label namespace bookshop istio.io/dataplane-mode=ambient is not that it is short. Plenty of catastrophic commands are short. It is that the cost of trying the mesh just went to approximately zero, and that changes what kind of decision adopting a mesh is.

With sidecars, enrollment is irreversible in every way that matters organisationally. Not technically — you can uninject and restart everything back — but you have already spent the political capital. You booked the window, you got four teams to agree, you restarted production. Nobody is going to say “actually, let us undo that and think about it for a month”. So the decision has to be made up front, with confidence, before you have seen a single byte of your own telemetry, on the strength of a vendor’s slide deck and a proof of concept in a namespace that does not matter. That is the worst possible moment to make it.

Ambient inverts that. You label a namespace, look at the mesh’s view of your traffic, find out what your services are actually doing to each other, and decide afterwards whether any of it is worth keeping. If it is not, you remove the label. The pods, which never noticed the mesh arriving, will not notice it leaving either.

That is not a small feature. It is the difference between a mesh being an architecture you commit to and a mesh being a tool you pick up. And it is worth being clear-eyed that the price was not zero: you now have a privileged DaemonSet on every node rewriting your pods’ network namespaces from the outside, and a second DaemonSet carrying every packet those pods send. That is a lot of trust, placed in a component your application cannot see and did not consent to. The next two chapters are about earning it — first by proving what ztunnel is actually doing to your traffic, and then by finding out exactly where its powers end.

Next: The mTLS You Didn’t Ask For, and the Identity You Didn’t Have

Comments