The Sidecars Came Out and Nobody Noticed. Once.

Three labels and a restart takes a namespace from sidecars to ambient, and my client saw 220 requests and 220 x 200 — which is one trial, on a cluster I control, and you should not plan a migration around it. Here is the migration, and the security control it silently takes away from you.

Here is the whole migration, and here is what the client saw while it happened.

The namespace is legacy. It runs catalog with sidecars — two pods, 2/2 each, an Envoy in both of them. A pod called prober sits in the same namespace hammering catalog in a loop, counting status codes, and it does not stop for the duration of the experiment — not while I relabel, not while I restart, not for a second. Then, underneath that live traffic, I take the sidecars out.

Three steps:

$ kubectl label ns legacy istio-injection-
$ kubectl label ns legacy istio.io/dataplane-mode=ambient
$ kubectl rollout restart deployment/catalog -n legacy

Remove the injection label. Add the ambient label. Restart, to drop the proxies that are already in the pods. And the pods come back like this:

catalog-5d9dc756d8-897dt   1/1   Running   0     31s
catalog-5d9dc756d8-xdjwc   1/1   Running   0     32s

1/1. One container. The Envoy is gone from inside the pod, and the pod is still in the mesh — encrypted, identified, and talking HBONE, which we can check directly:

NAMESPACE   POD NAME                    ADDRESS       NODE              WAYPOINT  PROTOCOL
legacy      catalog-5d9dc756d8-897dt    10.244.1.72   k8s-lab-worker    None      HBONE
legacy      catalog-5d9dc756d8-xdjwc    10.244.2.75   k8s-lab-worker2   None      HBONE
legacy      prober                      10.244.1.71   k8s-lab-worker    None      HBONE

And the client, which never stopped:

    220 200

Two hundred and twenty requests. Two hundred and twenty two-hundreds. Zero failures, zero connection resets, zero 000s — straight through a rolling restart that destroyed and recreated every pod serving that traffic.

I do not believe that number, and neither should you. Not because it is false. I ran it, that is the output, and I am not going to pretend otherwise. I disbelieve it because of what a single clean trial is worth — which is very close to nothing — and because I know something about that Deployment that makes a perfect score suspicious rather than reassuring. We will get to it in a moment, and it is the most useful paragraph in this chapter.

First, the mechanics.

What the three commands actually do

kubectl label ns legacy istio-injection- — note the trailing hyphen, which is kubectl’s syntax for remove this label. This stops the mutating webhook from injecting a proxy into pods created in this namespace from now on. It does exactly nothing to the two pods currently running, which still have their Envoys and will keep them until they die. Every word of the last chapter applies in reverse: the webhook fires on create, so un-labelling is a statement about the future.

kubectl label ns legacy istio.io/dataplane-mode=ambient — this enrols the namespace in ambient. It is the same label from chapter four of the foundations book, it needs no restart to take effect on ordinary pods, and it tells istio-cni-node to start redirecting the namespace’s traffic through the node’s ztunnel.

kubectl rollout restart deployment/catalog -n legacy — and this is the one that costs you something. The ambient label does not remove a sidecar. Nothing removes a sidecar from a running pod, because nothing can change a running pod: the proxy is in that pod’s spec, the spec is immutable, and the only way to get a pod without a proxy in it is to create a pod without a proxy in it. So you roll the Deployment, the webhook does not fire this time, and the replacement pods come up with one container.

That is worth stating plainly, because it is the sentence that determines how you plan the work: migrating to ambient costs you exactly one restart per workload — the last one you will ever do for the mesh. Adoption cost you a restart. Every upgrade after that cost you a restart. This restart is the one that ends the restarts. After it, mesh upgrades touch ztunnel and istiod and waypoints, and never your pods again.

Note the ordering, and note that it is deliberate. I removed the injection label before I added the ambient one, so that no pod was ever created while the namespace was asking for both data planes at once. The pods that were already running did spend a window with a sidecar inside them and an ambient label on their namespace, and the shop kept serving throughout that window — but I did not go and inspect whether ztunnel had captured those pods or skipped them. Istio documents that it will not apply ambient redirection to a pod that already has a sidecar; that is me relaying the docs, not reporting an experiment. Do the removal first anyway. It costs nothing and it means you never have to find out.

Now be suspicious of the 220

Here is why the clean run bothers me, and I would rather hand you the doubt than the trophy.

The Kubernetes book spent half a chapter proving that a bare rolling update drops requests. Not sometimes, not under load — three trials out of three, on an idle laptop cluster, with a rollout that never intentionally dropped below the desired replica count. The mechanism was a race: the pod is marked Terminating and SIGTERM is delivered to it before the endpoint removal has propagated to every kube-proxy in the cluster, so for a fraction of a second, rules still point at a socket that is already closed. The client gets a 000 — no status line, no response, nothing. The fix was a preStop hook that sleeps for five seconds and outlives the propagation delay.

And here is the thing I know about the Deployment I just rolled: catalog does not have a preStop hook. It never did. Go and read manifests/bookshop/catalog.yaml in the companion repo — the hook is on web, deliberately, because web is where the Kubernetes book demonstrated the fix. catalog was left bare on purpose, and bare is exactly the shape that dropped requests in that chapter.

So a rolling restart of catalog, under continuous traffic, ought to drop something. And it dropped nothing — not one failure in 220 requests.

The most likely explanation — and I want to be exact about the status of this, because it is a hypothesis and not a result — is that the mesh retried the connect failures for me. Istio’s Envoy proxies apply a default retry policy to connection-level failures, so a request that lands on a dead socket does not necessarily reach the client as an error; it reaches the client as a slightly slower 200, having been quietly tried again against a different endpoint. During the migration window there was an Envoy in the caller’s path (the sidecar in the still-old pods) and ztunnel in the path afterwards, and the failure mode we are talking about — a TCP connect to a closed socket — is precisely the class that a proxy retries most safely and most invisibly.

I did not isolate this. I did not compare against a no-mesh control on the same cluster on the same afternoon, I did not disable retries and re-run, and I did not count retries at the proxy. So the honest statement of what I know is this — and it is deliberately narrow: a sidecar-to-ambient migration of a two-replica service, under light continuous load, on a three-node kind cluster, produced no client-visible failures in one trial, and there is a plausible mechanism by which that result is flattering rather than typical.

That is a much less satisfying sentence than “the migration is seamless” — and it is the one I am going to stand behind.

What it means for you, planning a real migration on a service that people actually pay for:

  • Do not treat one green run as evidence. Not mine, and not your own. The canary chapter measured 93/7 on a 90/10 split and the Kubernetes book measured 85/15 on the same split — small samples are noisy, and a small sample of failures is the noisiest thing there is. If your migration produces zero errors on one namespace, that tells you about that namespace, at that hour, at that traffic level, and nothing else.
  • Measure, with something that counts. Fire real traffic and count status codes, in both directions, for the whole window — the way prober did. If you cannot count, you cannot claim.
  • Or just hold the traffic. For anything customer-facing, the cheapest correct answer is to stop being clever — drain the service, migrate it, put it back. Every argument for doing it live is an argument about convenience, and convenience is the first thing you trade away when the blast radius is customers.
  • And give your workloads a preStop hook before you migrate, not after. You are about to do a rolling restart of everything you own — which is the exact operation the Kubernetes book showed drops requests without one. A migration is a poor moment to discover that your service has been quietly dropping a fraction of a per cent of requests on every deploy for two years, and that nobody ever managed to attribute them.

Mixed mode is the whole reason this is possible

The thing that makes a namespace-by-namespace migration safe is that the two data planes coexist in one mesh, on one control plane, at the same time. This is not a transitional hack. It is the supported configuration, and I ran on it for the entire lab.

You already saw the evidence in the last chapter, but it is worth naming:

$ kubectl get mutatingwebhookconfiguration
istio-revision-tag-default
istio-sidecar-injector

That is a cluster installed with the ambient profile — and the sidecar injector is right there, armed, ready to serve any namespace that carries istio-injection=enabled. Meanwhile ztunnel runs as a DaemonSet on every node, serving any namespace that carries istio.io/dataplane-mode=ambient. One istiod configures both. One root CA issues identities to both. A sidecar pod in legacy and an ambient pod in bookshop ran simultaneously on this cluster for most of a day — calling each other, authenticated by the same SPIFFE identities, out of the same trust domain.

Which means the migration is genuinely incremental. You get to choose the increment.

Migrate a namespace at a time, and pick the first one for boredom rather than importance. You want a namespace with real traffic — so the experiment means something — and low stakes, so that a mistake means nothing. An internal tool. A batch namespace. Something whose worst outcome is a Slack message and not a page. Migrate it, leave it alone for a week, and work through everything below before you touch the next one.

Keep the two labels straight, because they are not opposites. istio-injection=enabled is a sidecar label. istio.io/dataplane-mode=ambient is an ambient label. They come from different eras of the project — one is Istio’s old convention, one is the new one — and nothing in Kubernetes stops you from putting both on the same namespace. Removing one is not the same act as adding the other. In a migration you will do both, and you should do them in the order above.

The part that will hurt you: your controls do not migrate

Now the warning, and it is the one I would put on the first slide if I only got one.

Migrating to ambient can silently remove controls you believe you still have. Not break them loudly. Not fail the apply. Remove them — while the YAML stays in Git, the object stays in etcd, kubectl get keeps printing a healthy row, and nothing anywhere raises its hand.

The reason is structural, and you can derive it for yourself from the previous book. In a sidecar mesh, every pod has a full Envoy in its path, whether it needs one or not. That Envoy speaks HTTP and parses every request. So every L7 feature you ever configured — an authorization rule about HTTP methods, a retry, a timeout, a header match, an egress restriction — had an enforcer, automatically, everywhere, for free, because you were already paying for one in every pod whether you used it or not.

In ambient you are not paying for that, which is the entire point — and the consequence is that an L7 control needs a waypoint or it has no enforcer at all. The authorization chapter is a whole essay on that exact failure, so I will not re-run it here. Just read it as a migration document rather than a security one: same policy, same caller, no waypoint, and DELETE /books comes back 200 where it used to come back 403.

And then there is the one that is worse — because it does not even have a waypoint-shaped fix.

REGISTRY_ONLY: a lock that stops working when you move house

Every Istio egress tutorial teaches this switch:

$ istioctl install --set profile=ambient \
    --set meshConfig.outboundTrafficPolicy.mode=REGISTRY_ONLY

Refuse any outbound connection to a destination that is not in the service registry. It is the standard lockdown. It is in your mesh config right now, if somebody sensible set your mesh up, and it is the reason your compliance document says egress is controlled.

Set it, on this cluster, the supported way. Confirm it landed. Then curl an external host from a pod in each data plane:

SIDECAR pod  (ns legacy,   istio-injection=enabled)  -> http://example.com : 502  BLOCKED
AMBIENT pod  (ns bookshop, dataplane-mode=ambient)   -> http://example.com : 200  NOT BLOCKED

Same cluster. Same setting. Same mesh config, sitting in the same ConfigMap, unchanged. The sidecar pod is blocked and the ambient pod is not.

The mechanism is not mysterious once you say it out loud, and it is the same shape as everything else in this book. outboundTrafficPolicy is enforced by the sidecar Envoy — it is in the pod, it sees the outbound connection, it checks the destination against the registry, and it refuses. That is why the failure is a 502: the Envoy answered. In ambient the path is pod → ztunnel → internet, and ztunnel is an L4 proxy that does not implement outboundTrafficPolicy. The waypoint does not save you either, because a waypoint handles L7 for mesh Services, and example.com is not a mesh Service, so there is no waypoint anywhere in that path.

Now assemble the story. A team runs a sidecar mesh with REGISTRY_ONLY, believing egress is locked down — and it is locked down, that 502 is real. They migrate to ambient, because it is cheaper and needs no restarts, both of which are true and both of which this book has spent two chapters telling them. And at the end of the migration their egress is open to the entire internet. No policy errored. No pod failed. No warning was printed anywhere. The setting is still in the config — spelled correctly, indented correctly, looking exactly as it did on the day it was reviewed and signed off.

Chapter seven takes that apart properly and is honest about what I did and did not verify: I verified the regression, not the remedies. What I want you to take from it here, in the migration chapter, is the general rule, because the specific bug is the least interesting thing about it.

A control is not a setting. A control is a setting plus something that enforces it — and migration changes the something.

The migration checklist, built around that

So do not write a migration plan that says “relabel, restart, done”. Write one that says: find out what our sidecars are actually enforcing, and prove each one still works afterwards.

1. Inventory the enforcement, not the intent. Go and list, per namespace: every AuthorizationPolicy — marking which ones carry L7 conditions (methods, paths, headers) as opposed to pure identity — every VirtualService and DestinationRule (retries, timeouts, outlier detection, connection pools), every mesh-wide setting in the istio ConfigMap, and every Sidecar resource. That list is your test plan. Every single thing on it is currently being enforced by an Envoy you are about to delete.

2. Sort it into three buckets.

  • Carries over for free: L4 and identity. mTLS, SPIFFE identities, and identity-only authorization policies (the selector:-bound, principals-only kind) are enforced by ztunnel on every node. They survive the move. This is most of your security value, and it is the part that gets cheaper.
  • Needs a waypoint: everything L7. L7 authorization, retries, timeouts, header routing, fault injection, request-level telemetry. These need an Envoy in the path, and in ambient the Envoy is a waypoint bound to the destination Service. Deploy the waypoint before you remove the sidecar, not after — that ordering is free and it means you never spend a minute in the unenforced state.
  • Does not carry over at all: outboundTrafficPolicy. See above. The reason it stops working is that example.com is not in Istio’s service registry, so nothing L7 is in its path to enforce anything. The obvious repair is therefore to put it in the registry — that is precisely what a ServiceEntry is for — and then an egress path can be given a waypoint and policed like any other Service. That is the shape of the remedy, and I did not run it. Chapter seven is scrupulous about this and I am not going to be less scrupulous here: I verified the regression, not the fix. Treat ServiceEntry, an explicit egress gateway, and a Kubernetes NetworkPolicy as the three things to evaluate, not as three things I can tell you work.

3. Test each control after the move. Actually test it. Not “check the YAML is still applied” — the YAML is always still applied, that is the whole problem. Send the request that should be denied and look at what comes back. Curl the external host that should be blocked. Poison a pod and see whether the outlier detection still ejects it. It takes ten seconds per control, and it is the difference between a locked door and a photograph of one.

4. Learn the new shape of “no” before you go looking for it. In a sidecar mesh a denial is a 403, because an Envoy composed it. In ambient an L4 denial is a connection reset — curl exit 56, http=000, no HTTP response at all — because ztunnel does not speak HTTP and has no way to compose a 403. Your dashboards count status codes; a dashboard that counts status codes will show you nothing at all when ztunnel refuses a connection. That is not a migration bug — but it is a migration surprise, and it will convince at least one person on your team that the policy has stopped working.

5. Run istioctl analyze in CI, and check .status.conditions. Istio 1.30 does report an L7 policy with no waypoint bound — WaypointAccepted: False, reason: AncestorNotBound — in the policy’s status and in istioctl analyze, which exits non-zero. It reports it in exactly the two places nobody looks, and in none of the places everybody looks. Make CI one of the places that looks.

Rolling back

If it goes wrong, the road back is the road you came on, walked backwards — remove istio.io/dataplane-mode, add istio-injection=enabled, roll the Deployment. The pods come back at 2/2 with an Envoy inside them, and every L7 control you used to have is enforced again: automatically, everywhere, because you are paying for one in every pod again.

I did not run the rollback. It is the same three commands and the same mechanism, and I am telling you that rather than showing you an output I do not have. What I will commit to is the shape of its cost, because that follows from the mechanism and not from an experiment: rollback is another full restart of the namespace. It is not a config flip and it is not instant. Which is worth knowing before you decide, at 2am, in the middle of an incident, that the safest thing to do is go back — because “go back” is another rolling update, with the same dropped-request race, on a cluster that is already unhappy.

That is an argument for migrating a boring namespace first, in the morning, with the traffic held. It is not an argument against migrating.

Final thoughts

The migration itself is almost insultingly small. Two labels and a rollout restart — you could do a namespace in the time it takes to read this paragraph, and mine came out the far side without dropping a request. The temptation, having watched that, is to schedule the whole fleet for a Thursday.

Do not. The relabelling is not the migration; the relabelling is the easy part that makes the migration look easy. The actual migration is the audit — the part where you find out that the egress lockdown you have been reporting to your auditors for two years is enforced by a proxy you are about to delete, and that nothing in the system will tell you when it stops working. That failure has no error message, no failed pod, no red line on a dashboard. Its only symptom is that a control which used to work does not work, and the only way to find out is to go and try the door.

Which is really the same lesson as the last chapter’s, arriving from the other direction. The last chapter was about believing a bug still exists after it has been fixed. This one is about believing a control still works after you have removed the thing that was enforcing it. Both are failures of the same kind — trusting a proposition about the system instead of asking the system — and the mesh is unusually good at producing them, because a mesh is precisely a layer of things that happen to your traffic without your application ever finding out.

So ask the system. Send the request. Count the codes. And when the count comes back 220 out of 220, be pleased for exactly as long as it takes to remember that you only ran it once.

Next: Gateway API Cannot Retry. So You Will Write Both.

Comments