The Job Completed. Nobody Told the Internet.

You do not get to choose the mesh you inherit.

The previous book built an ambient mesh from an empty cluster, and every chapter of it was written for someone who had that luxury. This one is about the world you actually walk into — where there is already a mesh, it is a sidecar mesh, it was installed in 2023 by someone who has since left, and the first thing anybody tells you about it is that it is terrible.

So let us start where you will start. Here is a namespace called legacy — the name is not a joke, it is what I called it in the lab — running a single service, and here is the shape of a pod in it:

catalog-5c68c7449b-cmb9g   2/2   Running   0     20s

2/2. Two containers, both ready, in a pod whose Deployment declares exactly one. You did not write the second one. You cannot find it in Git. It is a large C++ network proxy that terminates every connection into and out of that pod — and it got there because somebody, once, typed a label.

Everything you know about that second container, you probably learned from the internet. And a good deal of what the internet knows about it has been wrong for about two years.

This chapter does two things, and the second one is the point. First, it shows you exactly how the proxy gets into the pod — you cannot migrate away from a mechanism you cannot describe. Then it takes the most famous sidecar horror stories, the ones in every conference talk and every “why we moved off Istio” post, and runs them. Two of them are dead. The platform killed them. And if you are going to argue for ambient inside your own organisation — which this book thinks you should — then you had better argue for it on the grounds that are still standing, because somebody in that room has read the release notes and you have not.

Injection: a label and a webhook, and only one of them matters

The mechanism is simple enough to state in a sentence, and the sentence hides the whole problem.

You label a namespace:

$ kubectl label ns legacy istio-injection=enabled
namespace/legacy labeled

And from then on, every pod created in that namespace passes through a mutating admission webhook on its way through the API server — the same class of object the Kubernetes book took apart in the chapter on extending Kubernetes, where it turned out that a service mesh is, structurally, a control loop and a webhook wearing a trenchcoat. The webhook intercepts the CREATE request for the pod, rewrites the pod spec in flight, and hands the API server back a document with containers in it that you never wrote. The pod that lands in etcd is not the pod your Deployment described.

Here is the first thing worth knowing, and it is one you can check on your own cluster in ten seconds. The ambient install still ships the sidecar injector. This is not an either/or install:

$ kubectl get mutatingwebhookconfiguration
istio-revision-tag-default
istio-sidecar-injector

Two webhooks, on a cluster installed with --set profile=ambient, on which every namespace so far has been ambient. istio-sidecar-injector is sitting there — armed, idle, waiting for a namespace to carry the label. That is not an accident and it is not cruft. It is how one control plane runs both data planes at once, and it is the thing that makes the next chapter’s migration possible at all. Hold onto it.

Now the part that ruins people’s afternoons.

The label does nothing to the pods you have

Deploy the service into legacy before labelling anything. One container, as written:

catalog-86cd464f48-xb8mc   1/1   Running   0     1s

Now apply the label — the whole ceremony, the one every tutorial presents as “enabling the mesh” — and look at the pod again, immediately:

$ kubectl label ns legacy istio-injection=enabled
namespace/legacy labeled

catalog-86cd464f48-xb8mc   1/1   Running   0     6s

Still 1/1. Same pod, same name, same age ticking up, one container. The mesh is installed. The namespace is labelled. The webhook is running. And that pod is not in the mesh — is not going to be in the mesh, will never be in the mesh for as long as it lives.

The reason is not a bug and it is not a propagation delay you can wait out. The webhook fires on pod creation. That pod already exists. Its spec was admitted, validated, and written to etcd before the label was applied — and a pod’s spec is immutable in every field that matters. You cannot add a container to a running pod: not with kubectl edit, not with a patch, not with anything, because the kubelet was handed a manifest, built a sandbox from it, and the sandbox is what it is. The webhook is not a controller — it is not reconciling the world toward a desired state, it has no opinion about the pods that exist. It is a filter on a stream of CREATE requests, and a pod that already exists is not in that stream.

So there is exactly one way to get the proxy into that pod, and it is to destroy the pod and let a new one be admitted:

$ kubectl rollout restart deployment/catalog -n legacy

catalog-5c68c7449b-cmb9g   2/2   Running   0     20s

2/2. New pod name, new ReplicaSet hash, new spec, new proxy. The old pod is gone — and with it its connections, its warm caches, its JVM’s inlining decisions, and, if the rolling-update chapter taught you anything, a few of its in-flight requests.

Put that next to what enrolling a namespace in ambient cost — one label and nothing else. Six running pods: same names, same start times, zero restarts, still one container each, still serving 200s throughout. Not “a fast restart”. Not “a rolling restart with a good preStop hook”. No restart. The pods did not know it had happened.

That difference is structural, it is permanent, and it is the first of the two honest reasons to prefer ambient. It is not folklore — it is arithmetic about when a webhook fires.

Now open the pod. This is not the sidecar you were promised.

Here is where the chapter turns — and where the outline I wrote for this book turned out to be out of date, which is a thing I would rather tell you about than quietly fix.

Look at what is actually in that 2/2 pod:

containers:
  bookshop  (bookshop:v1)

init containers:
  istio-validation  (registry.istio.io/release/proxyv2:1.30.2-distroless)
  istio-proxy       (registry.istio.io/release/proxyv2:1.30.2-distroless)

Read it twice. There is one container in containers: — your application, and nothing else. The proxy is not there. The proxy is in initContainers, sitting under a second init container called istio-validation, whose job is to confirm that traffic redirection is set up before anything else in the pod runs.

But an init container runs to completion and then exits — that is the definition. So how is the pod 2/2, with a proxy that is supposed to run forever?

istio-validation: restartPolicy=
istio-proxy:      restartPolicy=Always

There it is. istio-proxy is an init container carrying restartPolicy: Always — which is to say it is a Kubernetes native sidecar. That is the exact mechanism the Kubernetes book demonstrated in the pods chapter, on a pod I wrote by hand, with a note attached saying it was the modern fix for a decade-old wart. Istio 1.30 uses it by default. The wart is fixed. Nobody sent a memo.

If you have not got that chapter in your head, the contract is this. A container declared under initContainers with restartPolicy: Always starts in init order — before any container in containers: — but it is not waited on for completion. Once it has started, the kubelet moves on to the next init container and eventually to your app. It then runs for the life of the pod, it is counted in the readiness fraction — hence 2/2, not 1/1, for a pod with a single app container — and it is terminated after the app containers on the way down. Prologue and show, as the Kubernetes book put it.

Two of the famous sidecar disasters are direct consequences of not having that contract. So let us go and see whether they still happen.

Horror story one: the app that starts before its proxy

The old failure was completely real. Containers in containers: start in parallel, with no ordering guarantee whatsoever — so your app comes up, dials http://catalog on the first line of its startup path, and the call fails, because the iptables rules are already redirecting that connection into an Envoy that has not yet received its configuration from istiod and has no idea where catalog is. The app crashes. Or, worse, it retries three times, gives up, and comes up in a degraded mode that nobody notices for a week.

The workaround every sidecar veteran can recite is holdApplicationUntilProxyStarts — a mesh-config flag that inserts a blocking wait so the app container does not begin until the proxy reports ready. It works, at the cost of adding the proxy’s startup time to every pod start in your fleet — and it is the sort of setting that ends up in a wiki page called “Things You Must Set Or Everything Breaks”.

With a native sidecar, the ordering hole is closed by the platform. The proxy is an init container. It starts before the app container. Not “usually”, not “if you set the flag” — the kubelet’s init sequence guarantees it, and the guarantee is a property of Kubernetes, not of Istio’s willingness to paper over its own problem. holdApplicationUntilProxyStarts is no longer the load-bearing hack it was.

Let me be precise about the limits of what I checked, because this is exactly the kind of claim that grows an extra clause every time somebody repeats it. What I verified is the field — istio-proxy is an init container with restartPolicy: Always — and the ordering consequence follows from the Kubernetes contract the previous book demonstrated on a hand-built pod. I did not race an application against the proxy’s config load and time the gap. I am not going to tell you that “started” and “has received its routing tables from istiod” are the same instant, because they are not necessarily the same instant and I did not measure it. What I will tell you is that the structural hole — app container and proxy container starting in parallel, decided by a coin toss — is gone, for the simple reason that they are no longer in the same list.

Horror story two: the Job that never ends

This is the big one. This is the story that has done more to sell ambient mode than any benchmark ever will.

You have a Job. It seeds the database, or reconciles a nightly report, or drains a queue. It runs in a mesh-enabled namespace, so it gets a proxy. Your container does its work and exits with a clean zero — and the pod sits there at 1/2, forever, because Envoy is a server and servers do not exit. The Job never completes. The CronJob that owns it never schedules cleanly, so Active pods pile up — one per interval, silently, week after week — until somebody opens the namespace in a month and finds four hundred of them.

Generations of engineers have bodged a curl -X POST localhost:15020/quitquitquit onto the end of their entrypoint script to make this go away. It is one of the most widely copy-pasted lines of shell in the industry, and it exists for one reason: two lifecycles were bolted together with no way to say which one was in charge.

So — run a Job in the sidecar namespace. It calls catalog through the mesh, which proves the proxy is genuinely in its path, and then it exits.

NAME   STATUS     COMPLETIONS   DURATION   AGE
seed   Complete   1/1           7s         40s

NAME         READY   STATUS      RESTARTS   AGE
seed-n6nzm   0/2     Completed   0          40s

$ kubectl logs -n legacy job/seed -c bookshop
job: called catalog through the mesh, exiting

Complete. 1/1. Seven seconds.

Look at the pod’s READY column: 0/2. Zero of two containers running — the app exited, and the proxy exited too — with STATUS: Completed. That fraction is the whole story in five characters. The Kubernetes book made a point of teaching you that READY counts containers rather than pods, and this is the payoff. On old Istio that column read 1/2 and stayed there until the heat death of the namespace. Here it reads 0/2 — because the kubelet terminates native sidecars after the app containers finish, which is the other half of the contract that restartPolicy: Always buys you.

The single most notorious bug in the history of the sidecar model is fixed — by Kubernetes, in the platform, and Istio picked it up as the default. Not worked around. Not mitigated with a flag. Fixed, at the layer where it was actually broken.

So how much of the folklore survives?

I want to be blunt here, because this is where a lot of otherwise-good technical writing goes bad. Ambient mode is better, and this book is going to spend thirteen more chapters saying so — which is precisely why it matters to be honest about the score. A book that repeats fixed bugs as current is worse than useless. It is a book that will get its reader laughed out of an architecture review the first time they cite it in front of somebody who has actually run 1.30.

So here is the ledger, with the dead items marked dead.

Dead — fixed by native sidecars. The Job that hangs at 1/2. The CronJob that silently backs up. The quitquitquit in the entrypoint. The app that starts before its proxy and fails its first outbound call. The startup-ordering flag in the wiki page. All of it is history on a current Istio and a current Kubernetes — and if the reason your organisation wants to leave sidecars is on this list, your reason has expired.

Still true, and structural. The restart. Everything above about the webhook still holds — adopting the mesh restarts every pod you own, leaving the mesh restarts every pod you own, and a CVE in the proxy (which will come, because it is a large C++ network program facing hostile traffic) restarts every pod you own. So does every minor upgrade. And Istio supports only two minor releases at a time, so “every minor upgrade” means roughly twice a year, forever. That is not a bug anybody can fix — it is what “a container in your pod” means.

Still true, and expensive. You pay for a full Layer 7 proxy in every pod, whether that pod speaks HTTP or not. Postgres gets an Envoy. Your batch worker gets an Envoy. The little Go process whose entire job is to write one metric gets an Envoy — holding a routing table that describes every service in the mesh, because by default every proxy is told about every destination just in case it needs one. That cost is per-pod, and it scales with the size of the mesh’s config rather than with anything your pod is actually doing.

Still true, and awkward. There is an Envoy in the path of every request into and out of every pod, so a bad config push is a fleet-wide event. Although this one cuts both ways, and the fair thing to say is that a broken sidecar breaks one pod while a broken ztunnel breaks one node’s worth of them. Blast radius is a shape, not a number.

And one genuine advantage of sidecars that I am not going to sell around. Every pod in a sidecar mesh gets the full Envoy feature set, right there, with nothing further to deploy — L7 authorization, header routing, retries, per-request telemetry, all of it, everywhere, by default. Ambient gives you L4 everywhere for almost nothing and asks you to deploy a waypoint where you want L7. That is a better deal for almost everybody, and it is also the deal that made the fail-open authorization footgun possible — because unbundling introduced a seam, and a policy can fall through a seam. Sidecars have no seam. It is worth knowing exactly what you are giving up.

What the sidecar actually costs, measured

Which brings us to the number, and the number is the honest argument.

kubectl top pod --containers, on the sidecar pod in legacy:

bookshop       cpu=0m     mem=2Mi
istio-proxy    cpu=3m     mem=40Mi

The application is two megabytes. The proxy standing next to it is forty. The thing you deployed — the thing that exists to serve books to customers, the thing with the business logic in it — is using five per cent of the memory in its own pod. The other ninety-five per cent is a network proxy you did not ask for, holding a routing table for a mesh you have never read.

And in the ambient namespace, on the same cluster, at the same moment, the same application:

catalog-6c4f598bdb-48wb4   catalog   cpu=1m   mem=4Mi
catalog-6c4f598bdb-ftbgd   catalog   cpu=1m   mem=5Mi
catalog-6c4f598bdb-jxpmc   catalog   cpu=1m   mem=5Mi

One container. No proxy in the pod at all. The mTLS, the SPIFFE identity, the L4 authorization — all of it is happening, and none of it is in there. It is in a ztunnel on the node, which that pod shares with everything else on that node.

Now do the multiplication, because that is where the number stops being a curiosity and starts being a budget. A hundred pods on a sidecar mesh is a hundred Envoys at roughly forty megabytes each — about four gigabytes of proxy, memory you are renting from a cloud provider, on which no customer request will ever be served. A hundred pods on an ambient mesh across three nodes is three ztunnels, plus a waypoint for each service that actually opted into Layer 7. Chapter eleven does that arithmetic properly — every figure measured on this cluster, none of them estimated — and it is the chapter to bring to the meeting where somebody asks what the mesh is even for.

That is the second honest reason to prefer ambient, and together with the restart it is the whole case. Two reasons. Both true. Both still true after the platform quietly fixed everything else.

Living with the mesh you inherited

You are not going to migrate on Tuesday. So here is what to know about the sidecar mesh you are running right now — and I am marking which of it I ran and which I am relaying, because that distinction is the only thing that makes a technical book worth its shelf space.

Injection is per-namespace, and it can be overridden per-pod. The namespace label is istio-injection=enabled; Istio also documents a pod-level annotation, sidecar.istio.io/inject, which opts an individual workload in or out against the namespace default. I ran the namespace label. The pod annotation I am relaying from the docs — confirm it before you build a policy on it, and confirm the interaction between the two in particular, because that is exactly the sort of precedence rule people get backwards.

The 2/2 is a readiness conjunction, and it will lie to you in a new way. A pod at 1/2 used to mean “the Job is stuck, the proxy will not die”. Now, with the proxy as a native sidecar, 1/2 on a long-running pod much more likely means your app is failing its readiness probe while the proxy is perfectly healthy — or the exact reverse. Read the fraction, then read kubectl describe. And remember that kubectl logs on a two-container pod needs -c, because it will not guess for you.

Anything you change about the proxy is a rollout. Proxy resource limits, proxy log level, the mesh-wide config the injector bakes into the pod spec at admission time — all of it is written into the pod at creation. So changing any of it means creating new pods, which means a rolling restart of everything affected. There is no “reconfigure the proxies in place” for the things that live in the pod template. Plan the change like a deploy — it is a deploy.

And test your Jobs on the version you are actually running. The fix above is real on Istio 1.30 with a Kubernetes new enough to support native sidecars, which here is 1.36. On an older Istio, or an older Kubernetes, or an Istio still pinned to the legacy injection template, you have the old behaviour and you still need the quitquitquit. The bug is fixed in the platform; it is not fixed in your cluster until you are on the platform that fixed it — and that sentence is the entire discipline of this book compressed into one line.

Final thoughts

I planned this chapter as a demolition. The outline said: show them the sidecar, show them the lifecycle bugs, show them the Job hanging at 1/2 until the end of time, and let the horror do the arguing for ambient. It would have been a good chapter — and a dishonest one — because when I went and ran it, the Job completed in seven seconds.

There is a general lesson in that, and it is not really about Istio. The folklore of an ecosystem is a snapshot of its worst year, preserved indefinitely by the people who left. Somebody hit the Job bug in 2021, wrote a furious post, gave a conference talk, and moved on. The post is still the top search result. The talk is still being cited in design docs by engineers who were not there. Meanwhile the fix landed quietly in an enhancement proposal that nobody outside sig-node read, Istio adopted it in a minor release, and the release note was one line long. The bug is dead — the story about the bug is immortal, and it will outlive all of us.

So do the migration. The next chapter does it, namespace by namespace, with traffic in flight the whole time. Do it because the restart is structural and will bill you twice a year for as long as you run a mesh — and because the proxy is twenty times the size of the application it is standing in front of. Those two facts are measured, they are true today, and nobody is going to fix them, because they are not bugs. They are the shape of putting a proxy in a pod.

But do not do it because you read somewhere that Jobs hang. Go and run one.

Next: The Sidecars Came Out and Nobody Noticed. Once.