Nobody Knows Who Called catalog

Here is the bookshop, fifteen chapters in. Three Go services, a Postgres, a small CRD controller we wrote ourselves, all running on a three-node cluster and all doing exactly what we told them to.

web  ──▶ catalog ──▶ postgres
 └─────▶ orders  ──▶ catalog

Somebody posts an order. orders accepts it, and before it will write anything down it validates the ISBN by calling http://catalog/books/978-0134494166. That call leaves the orders pod, resolves through cluster DNS to a ClusterIP, gets rewritten by the kube-proxy rules on the node, and arrives at one of the two catalog pods — picked at random, because a Service is not round-robin and never was.

It works. It has worked for fifteen chapters — no drama, no incidents, a green cluster. Now answer five questions about it.

Was that call encrypted? No. It was plain HTTP over the pod network. Anything that can see packets on the wire between two nodes — a compromised node, a misconfigured CNI, a sniffer in a debug pod with NET_RAW, your cloud provider’s VPC flow logs if you turn on the wrong option — can read the ISBN, the order, and the API key in the header. We terminated TLS at the gateway and then handed the request to a cluster in which nothing else is encrypted at all. The certificate protects the internet from your users. It protects nothing inside the cluster from anything else inside the cluster.

Did catalog know it was orders calling? No. It knows a TCP connection arrived from 10.244.2.39. That is a pod IP — an ephemeral, reused, entirely unauthenticated number. catalog cannot tell orders from web from a pod somebody kubectl runs in the default namespace, because there is nothing in that connection that says who is on the other end. We spent a whole chapter giving pods a cryptographic identity, and then used it exclusively to talk to the API server. Between your own services, nobody is who they say they are, because nobody says anything.

Could a random pod in another namespace call catalog directly? Yes, instantly, with curl. A ClusterIP is reachable from every pod in the cluster. The tool you reach for is a NetworkPolicy, and NetworkPolicy is a real and useful thing — but read what it actually matches on: pod selectors and namespace selectors, compiled by your CNI down to IP addresses and ports. It is a firewall. It knows that traffic came from an address that, at the moment the rule was compiled, belonged to a pod with a label. It has no opinion about who is calling and no way to form one, because there is no identity in the packet to have an opinion about. And it cannot say “orders may GET /books but may not DELETE /books”, because a firewall does not read HTTP.

Is catalog even working? You think so. Both pods are 1/1 Running and both are Ready — and we already know exactly how much that promises, because an entire chapter was spent finding out. Readiness is a promise about routing, not about health. A pod can pass its readiness probe on every single check and return a 500 to every single request. Kubernetes will keep it in the EndpointSlice and keep sending it a third of your traffic — forever, cheerfully — because the only question Kubernetes asked was “did the probe come back 200?” and the answer was yes.

And which service called which, how often, and how slowly? You have no idea. There is no request count, no latency distribution, no error rate, no dependency graph — nothing. kubectl logs gives you whatever your application chose to print, in whatever format the person who wrote it felt like, and joining the storefront’s logs to the catalog’s requires a correlation ID that nobody added. The cluster moves every one of these requests and records none of them.

Five questions, five answers you did not want. And not one of them is a bug — every component in that path is behaving exactly as designed. Kubernetes solved scheduling and lifecycle: put the containers somewhere, keep them running, give them a stable name, take them out of rotation when they die. It never claimed to solve what happens between them. That gap is the entire reason service meshes exist, and this chapter is about what one is, what it costs, and — this is the half most Istio introductions skip, and the half I want you to remember — why you very possibly should not install one.

The layer you keep writing by hand

Look at those five gaps again and notice that you already know how to fix all of them. You have known for years.

Encrypt the traffic: terminate TLS in the application, mint certificates, distribute them, rotate them. Identify the caller: sign a JWT, put it in a header, verify it on the other side. Authorize: check the claim against a policy. Retry the flaky call, with a budget and backoff and jitter so you do not turn a blip into a stampede. Time it out so a slow dependency does not exhaust your connection pool. Trip a breaker when the dependency is properly down. Emit a metric per request. Propagate a trace header. Report the latency histogram.

Every one of those is a solved problem, and every one of them is a library. Netflix wrote them into Hystrix and Ribbon, Twitter into Finagle, Google into Stubby and then gRPC. It works — genuinely, at enormous scale, it works — and it is still the right answer for a great many systems. Here is where it stops working:

The bookshop is three Go services because I wrote all three. Your company is not. It is a Java service from 2016 that nobody will touch, a Node BFF, two Python things the data team owns, a Rust rewrite of the thing that was too slow, and a vendored binary you do not have the source to. Now roll out mutual TLS — in the app, in five languages, across four teams, in a quarter. Now upgrade the retry library because it had a CVE, in all of them, coordinated, with every service shipping a deploy for it. Now ask the vendor binary to please emit a trace header.

And the policy is not yours. “No service may talk to Postgres except catalog” is a statement the security team wants to make, audit, and enforce. Under the library model, enforcing it means asking every team to import a library, configure it correctly, and then trusting them — forever, including the team that ships at 4pm on a Friday. There is no place to stand where you can say it once and have it be true.

That is the observation a mesh is built on. These concerns are not application logic — they are infrastructure wearing application logic’s clothes, and they ended up in your code because there was nowhere else to put them.

What a service mesh is

A service mesh moves that layer out of your process and into the network path, where you can configure it centrally and where it applies whether or not the application cooperates.

Concretely: a mesh puts a proxy in the path of every request between your services. The application still opens a plain HTTP connection to http://catalog — no SDK, no import, no code change, not one line — and the proxy intercepts it. On the way out it establishes a mutually-authenticated TLS session to the proxy on the other side, presenting a certificate that says I am the orders service account in the bookshop namespace. On the way in, the other proxy verifies that certificate, checks it against policy, applies whatever routing or retry or timeout rules exist, counts the request, records how long it took, and only then hands the plaintext to catalog, which has no idea any of this happened.

That gives you the standard two-part vocabulary, and it is worth getting straight now because the whole rest of this book leans on it.

The data plane is the proxies. They are in the path of your traffic, they touch every byte, and if they fall over your requests fall over with them. They do the encrypting, the authorizing, the routing, the retrying, the counting.

The control plane is the thing that configures the proxies. It watches the Kubernetes API — Services, EndpointSlices, pods, and its own custom resources — computes what each proxy in the fleet should be doing, and pushes that config out. It is also the certificate authority: it mints the workload certificates that give the data plane an identity to present. And it is not in your request path — a distinction worth holding onto. If the control plane dies, the proxies keep running on their last-known config and your traffic keeps flowing; you simply cannot change anything, and certificates eventually expire.

Everything a mesh does falls into five buckets, and it is worth naming them because they map one-to-one onto the five questions the bookshop failed:

• Identity — every workload gets a cryptographic name, derived from its Kubernetes ServiceAccount. This is the load-bearing one. Everything else is built on it.
• Encryption — mutual TLS between every pair of proxies, with certificates issued and rotated by the control plane, on a schedule you do not manage.
• Policy — allow and deny rules written against identities, not IP addresses, and evaluated at Layer 7 if you want them to be: this identity may call this service, with these HTTP methods, on these paths.
• Traffic control — routing by weight, by header, by path; timeouts; retries; circuit breaking and outlier detection.
• Telemetry — request counts, error rates, latencies, and a service dependency graph, for free, from the proxies, with no instrumentation in your code.

And the reason all five arrive together is that they are all the same trick: there is now a program in the request path that you control and your application developers do not have to know about. Once such a program exists, adding a metric to it is trivial — and so is adding a retry, a timeout, a header match, a deny rule. The mesh’s real product is the proxy’s position, not any individual feature. Every capability in the list above is a consequence of that position, which is why they all showed up at once and why the list keeps growing.

The honest half: what it costs you

Everything above is the pitch, and every vendor page on the internet stops there. Do not stop there.

You are running a control plane. It is a Kubernetes workload like any other — and it is now in your critical path for deployments, for certificate issuance, and for every config change. You have to monitor it, size it, alert on it, and upgrade it. Nobody at your company has done that before, and for the first six months, when something is weird, the mesh will be the first suspect and usually the wrong one.

You have put a proxy in the data path. Two of them, in fact — one on each side. Every request now traverses two extra processes, which costs latency (small; single-digit milliseconds is a fair mental model, and I did not measure it in this lab, so treat that as an order of magnitude and not a number). More importantly it costs you a failure domain that did not exist yesterday. When the proxy is misconfigured, your traffic is misconfigured — when the proxy has a bug, your traffic has a bug. You have added a component that can break your application while your application is perfectly healthy, and it will, at some point, and you will spend a long evening proving that it was not the app.

You have added a debugging surface. Before the mesh, “why is this call failing?” had a bounded set of answers — the app, DNS, the Service, the pod. After the mesh, the answer might be a proxy config that a control plane computed from a custom resource written by a team you have never met, and the way you find out is a CLI subcommand you have not learned yet. This book is largely a tour of that surface, which should tell you something about its size.

And you have signed up for a version treadmill. This is the cost nobody warns you about, and it is the one that actually bites.

Istio supports exactly two minor releases at a time. Not two years. Not “the last few”. Two — the current one and the one before it, with each release going end-of-life roughly six weeks after the release two ahead of it ships. In July 2026, as I write this, that means 1.30 and 1.29, and nothing else. There is no long-term support release. There is no version you can install and forget.

Sit with what that means operationally. Istio ships a minor roughly every quarter — so forever, for as long as you run this mesh, somebody on your team must upgrade a component that sits in the path of all of your production traffic, two or three times a year, on a clock set by somebody else. Miss two cycles and you are running an unpatched proxy in front of every service you own, which is precisely the position you installed the mesh to avoid being in.

It also has a nasty second-order effect on learning the thing. Anything you already know about Istio — and very nearly every tutorial, conference talk, and Stack Overflow answer you will find — is about a release that is now unsupported. The mesh you read about is not the mesh you can install. This book pins 1.30.2, and I will say this plainly rather than bury it: the version this book pins will fall out of support in about six months. That is not a flaw in the book — it is the shape of the project, and a reader who does not know it will end up running an unpatched mesh and calling it stable.

So: not yet

Here is the part I want you to be able to say out loud.

If you have three services, one team, one language, and no compliance requirement, you do not need a service mesh. Not “you could probably manage without one” — you do not need one, and installing one will cost you more than it returns.

Look at the bookshop, honestly. Three services. If I want mutual TLS between them, I control all three and I could do it in an afternoon. If I want orders to retry catalog, that is four lines in one Go file. If I want to know how many requests catalog served, I can add a counter. If I want to stop web from calling Postgres, there are two callers in the entire system and I can simply not write the call. The mesh’s whole value proposition is doing this uniformly across a fleet you do not control — and I control the fleet. It fits on a napkin.

The number that matters is not the service count on its own. It is teams, times languages, times things you cannot get everyone to do. A monolith plus two helpers, owned by one team, is a napkin. Forty services in five languages owned by nine teams — three of them contractors, one of them a vendor’s binary — is a mesh-shaped problem, and it is mesh-shaped precisely because you cannot hold a meeting big enough to solve it any other way.

Some other honest “not yets”:

• You have not got the basics right. If your services have no readiness probes, no resource limits, and no preStop hooks, a mesh will not save you — it will hand you a second, more sophisticated way to be broken, and it will make the first one harder to see. Fix the Deployment before you buy the mesh.
• You cannot yet afford to operate it. A mesh you install and never upgrade is worse than no mesh — it is an unpatched, privileged, cluster-wide network component with your name on it. If nobody owns the upgrade, do not install it.
• You want it for exactly one feature. People install Istio for the dashboard. If what you want is metrics, buy metrics — and if what you want is a canary at the edge, Gateway API already does weighted routing, and you have already done it.

And: yes, actually

There is a real other side, and I would not be writing eleven more chapters if there were not. A mesh is the right call when one of these is true, and the more of them are true the more obviously right it gets.

Zero trust, or an auditor. You need every service-to-service call encrypted and every caller authenticated — and you need to prove it. This is the single strongest case, and it is not really about the encryption. It is about the identity. Once every workload presents a certificate, “which services may call the payments service?” becomes a question with a written, enforceable, auditable answer, instead of a NetworkPolicy that matches on IP ranges and a shared belief that nobody has done anything silly. If a compliance regime is asking you for encryption in transit inside the cluster, a mesh is close to the only answer that does not involve rewriting every service you own.

Enough services that policy cannot be a social process. When the answer to “does this service retry?” is “depends who wrote it” — and the answer to “can I see the call graph?” is “no” — and there are enough services that fixing either by asking politely is a multi-quarter programme, that is the mesh’s home ground. It replaces N conversations with one config.

Progressive delivery by request, between services. We did a canary at the edge with Gateway API, and it was genuine: 90/10 weights, about 85/15 measured over a hundred requests, no pod restarted. But that was at the front door. Inside the cluster, when orders calls catalog, there is no gateway — there is a ClusterIP and a random pick over the EndpointSlice. Your only lever on that split is pod count. To send 10% of catalog traffic to a new version you have to run nine old pods and one new one, and to change the ratio you have to change the number of pods — a blunt, slow, expensive instrument that welds your traffic policy to your capacity planning. A mesh moves the split from pods to requests: 90/10 by weight, or “every request carrying this header”, between any two services, without scaling anything.

Failures that Kubernetes structurally cannot see. This is my favourite argument, and the one I will spend a whole chapter on. Take a catalog pod and break it in the specific way that matters — make it fail every request while its readiness probe keeps returning 200. Kubernetes sees three healthy pods. The Service keeps all three in rotation. Roughly one request in three fails, forever, and nothing in Kubernetes will ever notice, because from the cluster’s point of view the pod is doing everything it was asked to do. Thirty requests, no outlier detection: twenty 200s and ten 500s. Add outlier detection — a few lines of config, nothing touched in the app, no restart — and the same thirty requests come back thirty 200s and zero 500s, with the bad pod still Running, still Ready, still sitting in the EndpointSlice. The mesh saw what the probe could not, because the mesh watches the responses and the probe only ever watched the probe.

That last one is the shape of the whole argument. A mesh does not give you a capability Kubernetes forgot — it gives you a vantage point Kubernetes does not have. A place in the request path that can see what actually happened, and act on it.

The bet this book makes

There is one more thing to say before we go any further, because it changes the cost side of the ledger and it is why this book exists in the shape it does.

The costs I listed above — the proxy in the path, the memory, the restarts, the blast radius — were all written down in the era when a mesh meant a sidecar: an Envoy proxy injected into every single pod you run. That model is real, it is mature, and it is what every tutorial you will find online assumes. It also means that adopting the mesh requires restarting your entire fleet, that a pod which needs no HTTP-level features pays for an HTTP proxy anyway, and that upgrading the mesh means restarting the fleet again.

Istio now has a second data plane — ambient mode — which splits the job in two: a lightweight per-node proxy that gives every pod identity and mutual TLS with no sidecar and no restart, and an optional Layer 7 proxy you deploy only where you actually need Layer 7. It genuinely changes the arithmetic. In this lab, enrolling the entire bookshop into the mesh took one kubectl label and produced zero pod restarts — same pod names, same start times, still one container each — while the shop kept serving 200s throughout.

This book leads with ambient, which is a deliberate bet against nearly every other Istio resource you will read. The next chapter is that argument in full, made honestly, including the things ambient is genuinely worse at.

But note what ambient does not fix. It does not remove the control plane. It does not remove the failure domain — it moves it from the pod to the node, which is a trade rather than a deletion. It does not shrink the debugging surface, and it does not touch the two-release support window by so much as a day. Ambient makes a mesh dramatically cheaper to adopt; it does not make one cheaper to own. “Not yet” remains the right answer for a great many people, and I would rather you reach it here than six months into an adoption.

Final thoughts

The five questions at the top of this chapter all have the same answer, and it is not “Kubernetes is missing a feature”. It is that Kubernetes drew its boundary at the pod. It will schedule your containers, keep them alive, give them stable names, and route to the ones whose probes are passing — and at the instant your process opens a socket to another service, the cluster’s interest in what happens next ends. Everything in that gap is your problem, and for most of the industry’s history the way we solved it was to import a library and hope.

A mesh is what you build when hope stops scaling. It is not a product you add to a system — it is a layer you insert underneath one, and inserting a layer under a running system is one of the more consequential things an engineer can do. The reason to be slow about it is not that it is hard to install; you will install it in the third chapter, in about ninety seconds. It is that once it is there it sits in the path of every request you serve, and it now needs an owner, an upgrade cadence, and somebody who understands it at 3am.

So here is the test I would apply, and the one I would want applied to me. Can you name, right now, the specific thing you cannot do today? Not “we should have mTLS” — which call is unencrypted, and who is asking. Not “we want observability” — which question about your system are you currently guessing the answer to. If you can name it, and the answer is not obviously a library import, a NetworkPolicy, or a readiness probe you never wrote, then you have a mesh-shaped problem and the rest of this book is for you.

And if you cannot name it, say “not yet” out loud, close the tab, and go fix your readiness probes. That is not a joke and it is not a cop-out. It is the highest-value thing most teams reading this could do this week, and no mesh will do it for them.

Next: A Proxy in Every Pod Is a Restart in Every Pod