The Proxy Is Twenty Times the Size of Your App

Two lines from kubectl top pod --containers, taken against a pod in the legacy namespace — the one running the classic sidecar data plane:

POD                        NAME           CPU(cores)   MEMORY(bytes)
catalog-5c68c7449b-cmb9g   bookshop       0m           2Mi
catalog-5c68c7449b-cmb9g   istio-proxy    3m           40Mi

bookshop is the application — the reason the pod exists, the thing that was written, reviewed, tested and deployed, and the thing the business is actually paying for. It is using two megabytes.

istio-proxy is the sidecar. Nobody asked for it, nobody wrote it, and it arrived because a namespace was labelled. It is using forty megabytes.

The proxy is twenty times the size of the application it wraps.

This chapter is the numbers — not the vendor’s numbers, not a benchmark from a conference slide, not “typically around”, but the figures this cluster actually reported, per container, for every component of both data planes. They are the reason to trust the rest of this book: they are the only numbers in it that anybody can check against their own cluster in about ninety seconds, and they settle an argument that is usually conducted entirely on folklore.

They also come with an honest boundary around them, and I am going to draw that boundary before I show you anything else, because a measurement without its conditions attached is just a rumour with a decimal point.

What these numbers are, and what they are not

Every figure in this chapter comes from kubectl top, per container, on the three-node kind cluster this whole book runs on. The bookshop is small. Its traffic during measurement was light — I was curling it, not load-testing it. The mesh config is small: a handful of services, a handful of policies, one waypoint.

So:

These are idle-ish numbers on a small mesh. They are a floor, not a forecast. A busy Envoy holding a big mesh config uses considerably more — for reasons I will come to — and it would be dishonest to hand you 40Mi as though it were a capacity plan.

I did not benchmark under load. I did not measure latency. No p50, no p99, no requests-per-second, no CPU-under-saturation curve. If you want a latency number from me, I do not have one — and I am not going to reach for somebody else’s and present it as though I had watched it happen. What I will do, later in the chapter, is reason from the architecture about where the latency has to come from, and label that reasoning as reasoning.

What these numbers are is the thing most cost discussions skip entirely — a like-for-like, same-cluster, same-application, same-afternoon comparison of the two data planes, with the components broken out so you can do the arithmetic yourself. That arithmetic turns out to be decisive, and it is decisive by two orders of magnitude. That is a wide enough margin that the fuzziness in any individual figure cannot change the conclusion, which is the only reason I am comfortable drawing one from a laptop.

The sidecar pod, measured

Here is the whole sidecar cost, from the legacy namespace where chapter one installed the classic data plane:

bookshop       cpu=0m     mem=2Mi
istio-proxy    cpu=3m     mem=40Mi

Forty megabytes and 3 millicores of CPU, at rest, doing nothing, for every pod in the mesh.

The CPU figure is the less interesting one, and it is worth putting aside quickly. 3m is three thousandths of a core — the proxy is essentially asleep, ticking over its stats and its xDS connection. Under load it will not be asleep, that is where the real CPU cost lives, and I did not measure it. Note the caveat and move on.

The memory figure is the one that decides architectures, and the reason is simple — it does not go away. It is not a burst, it is not a cache you can evict, and it is not proportional to traffic. It is the resident set of a full Envoy: an entire L7 proxy, with its listeners, its clusters, its routes, its TLS contexts and its stats registry, sitting there whether the pod serves ten requests a day or ten thousand a second.

And critically, it is sitting there once per pod. That is the sentence to hold on to, because everything else in this chapter is a consequence of it.

The ambient pod, measured

Same application. Same cluster. Same afternoon. The bookshop namespace, running ambient:

catalog-6c4f598bdb-48wb4   catalog   cpu=1m   mem=4Mi
catalog-6c4f598bdb-ftbgd   catalog   cpu=1m   mem=5Mi
catalog-6c4f598bdb-jxpmc   catalog   cpu=1m   mem=5Mi

Three pods, three containers. There is no second container. No proxy in the pod, no istio-proxy row, nothing to list — because in ambient mode, as the foundations book established, the pod’s traffic is intercepted by the node’s CNI and carried by a node-level proxy that lives somewhere else entirely.

The mesh is emphatically on. These pods have SPIFFE identities, they are speaking mTLS over HBONE, they are subject to authorization policy, and they are reporting telemetry — every capability from the foundations book is live on them. And the per-pod cost of all of it, measured, is zero containers and zero megabytes, because the cost has been moved somewhere you pay for it once instead of N times.

(The app containers read 4–5Mi here against 2Mi in the sidecar pod. I would not read anything into that — it is the same image, and a two-megabyte delta at this scale is noise, not a finding.)

Here is where you actually pay.

The shared cost

istiod-644c69ddc-8zxdd      cpu=2m   mem=74Mi
ztunnel-6tqhs               cpu=1m   mem=8Mi
ztunnel-lp772               cpu=4m   mem=4Mi
ztunnel-t7dn4               cpu=5m   mem=9Mi
waypoint-6b8b5b8b9c-vnm6m   cpu=4m   mem=51Mi

Four things, and each one scales along a different axis. That is the entire insight of this chapter, so read the list slowly.

istiod: 74Mi. One control plane, for the whole mesh. This is the largest single number on the page and it is also the least interesting — because there is exactly one of it. It scales with the size of your config and the number of proxies it has to push to, not with your pod count directly and certainly not linearly. You pay it once.

ztunnel: 4Mi, 8Mi, 9Mi. Three node proxies on three nodes — and the whole L4 data plane of the mesh, meaning mTLS, identity, HBONE tunnelling and L4 authorization, comes to twenty-one megabytes across the entire cluster. That is a rounding error. It is half of one sidecar. The variation between the three is presumably a function of how many workloads each node happens to be carrying, though I did not chase it down, and at these magnitudes it does not matter.

Waypoint: 51Mi. This is an Envoy — a real one, the same lineage as the sidecar — and its footprint says so. It is the most expensive per-instance thing in the ambient data plane, and it costs more than a sidecar does. But note what it is not: it is not per-pod. It is per-Service, and only for the Services where you deployed one — which, per the authorization chapter, is the set of Services where you needed to enforce something about HTTP rather than something about identity.

Sidecar Envoy: 40Mi, per pod, unconditionally. Which brings us to the arithmetic.

The arithmetic that decides the architecture

Take a mesh with 100 pods on 3 nodes. That is a modest production service — a dozen microservices with a handful of replicas each — and it is the size where these questions start getting asked.

Sidecar data plane. One Envoy in every pod. 100 pods means 100 Envoys means:

100 × 40Mi = 4,000Mi ≈ 4 GB of proxy memory

Four gigabytes of RAM, cluster-wide, spent on proxies — not on your application, on the sidecars wrapping it. That is a meaningful fraction of a node, and none of it is doing your work. It is doing the mesh’s work one hundred separate times, in one hundred separate copies of the same process, holding one hundred separate copies of substantially the same configuration.

Ambient data plane. One ztunnel per node. 100 pods on 3 nodes means 3 ztunnels means:

3 ztunnels ≈ 21 MiB

Twenty-one megabytes. Plus a waypoint at 51Mi for each Service that actually opted into L7 — and if that is three of your Services, that is another 153Mi, which puts the entire data plane under 200 MiB.

Four gigabytes against about a fifth of one. That is not a tuning difference and it is not an efficiency gain — it is two orders of magnitude, and it comes from changing what the cost scales with:

	scales with	100 pods / 3 nodes
sidecar	pod count	100 Envoys ≈ 4 GB
ambient L4	node count	3 ztunnels ≈ 21 MiB
ambient L7	services that opted in	51Mi each, for those services only

Now double the fleet. Two hundred pods on the same three nodes — you scaled the replicas, not the cluster. The sidecar mesh doubles to eight gigabytes of proxy. The ambient mesh’s L4 cost does not change at all, because you did not add a node. And its L7 cost does not change either, because you did not add a service that needs L7 — the existing waypoints simply carry more traffic.

That asymmetry is the honest, load-bearing, technical reason to prefer ambient. It is not folklore, it is arithmetic, and it is reproducible on your own cluster this afternoon.

And it is not the folklore reason

I want to nail this down, because the case for ambient is usually made badly, and the bad version of it is now out of date.

For years the argument against sidecars was about lifecycle — the proxy that starts after the app and lets the first requests fail; the proxy that will not exit, so your Job hangs at 1/2 forever and your CronJob quietly accumulates zombies; the holdApplicationUntilProxyStarts incantation everyone copy-pasted without knowing why.

Chapter one went and checked, and those arguments have largely expired. Istio 1.30 injects istio-proxy as a Kubernetes native sidecar — an init container with restartPolicy: Always — so the platform now guarantees the ordering, and a Job with a sidecar in it completes. I ran it: seed Complete 1/1, the pod 0/2 Completed. The most notorious sidecar bug in the history of the technology is fixed, and it was fixed by Kubernetes rather than by Istio.

So if you are arguing for ambient on the basis of lifecycle horrors, you are arguing from a version of the world that has moved on — and somebody in the room will know it, and you will lose the argument for entirely the right reasons.

Argue from the 40Mi instead. The proxy is twenty times the size of the app, there is one of them in every pod, and that is the part that has not been fixed — because it cannot be fixed. It is not a bug, it is what the architecture is. A sidecar mesh means an L7 proxy per workload, and an L7 proxy costs what an L7 proxy costs.

Why a sidecar’s memory is not about your traffic

Now the caveat that makes the 40Mi worse rather than better in a real deployment, and it is the thing that most surprises people the first time they hit it in production.

An Envoy’s memory does not primarily scale with the traffic it serves. It scales with the amount of mesh configuration it has to hold.

Think about what the sidecar next to your catalog pod has been told. By default istiod pushes it configuration for everything it might conceivably need to talk to — the clusters, the endpoints, the routes and the TLS contexts for every Service in the mesh — because the control plane cannot know in advance which of them catalog will actually call. In a ten-service bookshop that is nothing. In a nine-hundred-service enterprise mesh, every single sidecar in the fleet is holding a map of all nine hundred services and all of their endpoints — in every one of your thousands of pods — and every time an endpoint anywhere changes, that change is pushed to all of them.

This is why large sidecar meshes reach for the Sidecar resource — a config object whose entire purpose is to tell istiod “this workload only ever talks to these namespaces, so stop sending it the rest of the universe.” Trimming the config trims the proxy’s memory, and on a big mesh that is not a micro-optimization. It is the difference between a fleet that works and one where a routine config push causes a cluster-wide memory spike.

I did not measure this — my mesh is small, and a 40Mi proxy on a ten-service mesh tells you nothing reliable about the same proxy on a nine-hundred-service one. But the mechanism is well documented, the Sidecar resource exists precisely because of it, and the implication is uncomfortable: the number in this chapter is a floor that grows with your organisation, not with your traffic. The bigger the mesh gets, the worse the per-pod arithmetic becomes — which is exactly the wrong direction for a cost to scale in, since the mesh is largest at precisely the moment you can least afford to multiply anything by the pod count.

Ambient sidesteps that particular trap at L4, because ztunnel is not holding an Envoy config at all — it holds an identity-and-endpoint view of the mesh, and it is one process per node rather than one per pod. It does not sidestep it for waypoints, which are Envoys and are subject to exactly the same physics. But there are a handful of waypoints, not thousands, and a handful of proxies whose config grows is a problem you can afford.

The latency cost, reasoned honestly

Here is where I have to be careful, because latency is the number everybody wants and it is the number I do not have.

I did not measure latency. I have no p50 and no p99 for either data plane, under any load, and I will not quote one. If you find a service mesh comparison that hands you a confident p99 without telling you the request rate, the payload size, the CPU allocation and the config size, treat it the way you would treat a benchmark in a press release.

What I can do is reason from the architecture, and label it as reasoning.

A sidecar adds two proxy hops. The request leaves your app, passes through the sidecar in its pod, crosses the network, passes through the sidecar in the destination pod, and arrives. Two Envoys — both on the same host as the process they serve, so there is no extra network round trip, but there are two full L7 parses on every request whether you needed L7 or not.

Ambient at L4 adds two ztunnel hops. Out through the ztunnel on the source node, across the network over HBONE, in through the ztunnel on the destination node. Also two hops, also node-local — but the proxies are doing less. ztunnel is L4, it is written in Rust, and it does not parse HTTP at all. In principle that ought to be cheaper than an Envoy. I did not measure it, so “in principle” is exactly where I am leaving it.

Ambient at L7 adds a waypoint, and that one is a real network round trip. This is the part people underestimate, and it is the honest cost of the ambient architecture. A waypoint is not in your pod and it is not on your node — it is a Deployment, sitting somewhere in the cluster, and traffic to a waypointed Service is routed to the waypoint and then on to the destination. That is a hop across the pod network that a sidecar mesh simply does not make, because with a sidecar the L7 proxy was already in the pod.

So the shape of the trade, reasoned rather than measured: ambient is probably cheaper than sidecars on the L4 path, and probably more expensive on the L7 path — and how much of your traffic is on each path is a question about your own topology that nobody, including me, can answer for you.

If your service budget is in single-digit milliseconds, measure before you adopt. Not “read a blog post about latency” — which is, I notice, what you are currently doing. Measure: with your payloads, at your request rate, on your hardware, with your waypoints where you would actually put them. A mesh puts proxies in the path of 100% of your traffic, and the trouble with finding out afterwards is that by then it is already in the path of 100% of your traffic.

The cost that is not in any of these numbers

And now the part of the bill that never shows up in kubectl top, and which in my experience is the part that actually hurts.

You now run a control plane. istiod is 74Mi of memory and an unbounded amount of somebody’s attention — it needs monitoring, it needs alerting, it needs somebody who knows what it does when it is unhealthy, and it is in the certificate-signing path of every workload you own.

You are on a version treadmill, and it moves fast. Istio supports exactly two minor releases at a time, as the installation chapter laid out at some length. What you pin today is unsupported in about six months. That is not a one-off migration cost — it is a permanent, recurring, twice-a-year obligation to upgrade a component sitting in front of every request you serve, and it is the entire subject of the next chapter, because it is the operational cost most teams quietly fail to pay.

You have a new failure domain. Before the mesh, a request from web to catalog failed because web was broken, or catalog was broken, or the network was broken. Now it can also fail because a policy landed on nothing, a waypoint is not where you thought it was, a retry hid a real error, an outlier detector ejected a healthy pod, or a certificate did not renew. Every one of those is a class of outage you did not previously have — and you acquired the whole set on the afternoon you labelled the namespace.

And you have a debugging surface that did not exist. Every failure in the paragraph above is invisible to your application, invisible to your application’s logs, and visible only through a set of tools your on-call engineer has probably never used — and will first meet at three in the morning. That is the chapter after next, and it is the one I would make people read before the mesh goes to production rather than after.

None of that is denominated in megabytes. All of it is denominated in engineer-hours — which are considerably more expensive than RAM, and which, unlike RAM, you cannot buy more of by adding a node.

Final thoughts

There is a way of reading this chapter as an advertisement for ambient, and the numbers do point that way — 4 GB against 21 MiB is not a close call, and if you are choosing a data plane today for a mesh you are going to grow, choose the one whose cost scales with your nodes rather than your pods. That conclusion is solid, it is measured, and it is the conclusion I would defend.

But it is not the first question, and I do not want the arithmetic to smuggle the first question past you.

The first question is whether you should have a mesh at all — which is where the foundations book opened, deliberately, before anybody had installed anything. It said that for a great many teams the honest answer is “not yet”, and nothing I measured here changes that. Twenty-one megabytes of ztunnel is cheap. A control plane in the certificate path of every service you own, a support window shorter than your planning cycle, a new class of silent failure, and a debugging surface nobody on the team has practised on — those are not cheap, and they do not become cheaper because the data plane became efficient.

What the numbers do is make the tradeoff legible. Before you had them, the argument was conducted entirely in adjectives — sidecars are heavy, ambient is light, meshes are complex. Now you can say something falsifiable: our mesh will cost four gigabytes of proxy memory as sidecars, or two hundred megabytes as ambient, and either way it will cost one control plane, two upgrades a year, and an on-call rotation that has to learn istioctl. Put that sentence in front of the people who have to approve it — and let them decide with the real number in their hand rather than an adjective.

That is what an engineering decision looks like — not a preference, but a price, stated out loud, by someone who went and checked. The mesh is worth it when what it buys you is worth more than that. And if you cannot say what it buys you in a sentence as concrete as the one above, the answer is still “not yet”, and there is no shame in it.

Next: Your Mesh Has Six Months to Live