The mTLS You Didn't Ask For, and the Identity You Didn't Have
HBONE, SPIFFE, a 24-hour certificate, and the discovery that every service in the bookshop had exactly the same identity — which made mutual TLS a secure channel between two things we could not tell apart.
The last chapter ended with a claim I have not yet earned: that after one kubectl label, every connection inside the bookshop is mutually authenticated and encrypted. Nothing about the pods changed. No certificate was mounted. No application was told anything. And yet the claim is that a TCP connection your Go code opened with a bare http.Get("http://catalog/books") — no TLS, no client cert, no crypto/tls anywhere in the binary — is now carrying a cryptographic proof of which workload sent it.
That is either true or it is marketing, and the only way to find out which is to go and look at the certificates.
This chapter goes and looks. It is also the chapter where the mesh quietly hands you a report card on a decision you made months ago and did not think of as a decision — and the grade is not good.
First: is it actually on?
Ask ztunnel what it thinks of each workload:
$ istioctl ztunnel-config workload
NAMESPACE POD NAME ADDRESS NODE WAYPOINT PROTOCOL
bookshop catalog-8c6d8db8c-2kd49 10.244.1.57 k8s-lab-worker None HBONE
bookshop catalog-8c6d8db8c-7fp5c 10.244.2.38 k8s-lab-worker2 None HBONE
bookshop orders-db6c9c86c-blbfb 10.244.2.39 k8s-lab-worker2 None HBONE
bookshop postgres-84bfd89b75-lmppf 10.244.2.42 k8s-lab-worker2 None HBONE
bookshop shelf-controller-6f9877c9bd-csd2h 10.244.2.45 k8s-lab-worker2 None HBONE
bookshop web-57b4dcbd6b-92qtr 10.244.1.59 k8s-lab-worker None HBONE
bookshop web-57b4dcbd6b-gqlrz 10.244.2.44 k8s-lab-worker2 None HBONESeven workloads, seven HBONEs. That column is the whole answer to “is mTLS on”, and it is worth understanding why, because HBONE is not a synonym for “encrypted” — it is the name of a specific and slightly unusual thing.
What HBONE actually does to your bytes
HBONE is HTTP-Based Overlay Network Environment, which is a name that tells you nothing, so ignore it and look at the mechanism.
Your web pod, sitting on k8s-lab-worker, opens a TCP connection to orders at 10.244.2.39. That is what it thinks it did, and from inside the container that is exactly what happened — a socket, a SYN, a plain connection to a plain address.
What actually happens on the wire is this. The redirection rules istio-cni-node installed in that pod’s network namespace steer the connection to the local ztunnel. ztunnel looks up the destination — it knows the full workload table you just printed, because istiod pushed it — and finds that 10.244.2.39 is orders, on node k8s-lab-worker2. It then opens a mutually authenticated TLS connection to the ztunnel on that node, and inside that TLS session it speaks HTTP/2, and inside HTTP/2 it issues a CONNECT to the destination address. CONNECT is the verb HTTP has always had for “stop being HTTP and start being a pipe” — it is what your browser does to reach an HTTPS site through a proxy. The remote ztunnel accepts it, opens a plain TCP connection to orders on its own node — and from that moment the two ztunnels shovel bytes between the two streams.
So: your TCP connection is tunnelled inside an HTTP/2 stream inside mutual TLS between two node proxies. That is the whole design, and three properties fall out of it that are worth naming.
It is protocol-agnostic. The tunnel carries bytes. It does not care whether those bytes are HTTP, gRPC, Postgres wire protocol, Redis, or a raw binary stream you invented. This is why postgres shows up in that table with HBONE next to it, exactly like the HTTP services do, and why it works. A sidecar-era mesh made you think hard about protocol detection and port naming for exactly this reason — the HBONE tunnel does not have an opinion.
One TLS session can carry many connections. HTTP/2 multiplexes streams, so the ztunnel-to-ztunnel session between two nodes is established once and then carries every connection between every pod on node A and every pod on node B, each as its own stream. You are not paying a TLS handshake per request, or even per connection.
The identity is in the TLS layer, not the tunnel. The CONNECT tells the remote ztunnel where to deliver the bytes. The client certificate on the TLS session tells it who is sending them. Those are two different questions answered by two different layers, and the second one is the one this chapter is about.
I did not packet-capture the tunnel, so treat the port numbers and frame layout as documented rather than measured; what I ran is the HBONE column, and what I am about to run is the certificate.
The certificate, and the trap in front of it
Ask for it the obvious way:
$ istioctl ztunnel-config certificate
Error: ztunnel pod name or --node must be setThat error is not a bug and it is not being difficult. It is telling you something structural that you should internalise before you work around it.
There is no such thing as “the mesh’s certificate”. Certificates are per-workload — they are held in memory by the ztunnel on the node where that workload runs, and different nodes hold different certificates because different workloads are scheduled on them. istioctl ztunnel-config talks to one ztunnel and dumps its view, so it needs to know which one. There is no sensible default, and Istio declines to invent one. Compare with the workload table above, which happily printed with no target — that table is cluster-wide state pushed to every ztunnel, so any ztunnel can answer it. The certificates are not. The command’s inconsistency is a faithful reflection of a real asymmetry in the system.
So name a node:
$ istioctl ztunnel-config certificate --node k8s-lab-worker
CERTIFICATE NAME TYPE STATUS VALID CERT SERIAL NUMBER NOT AFTER NOT BEFORE
spiffe://cluster.local/ns/bookshop/sa/default Leaf Available true d87fdb3c02aa33939819e7623227cd5d 2026-07-15T02:49:49Z 2026-07-14T02:47:49Z
spiffe://cluster.local/ns/bookshop/sa/default Root Available true 5721f66243f0a2dba12e34ed34b869ad 2036-07-11T02:49:18Z 2026-07-14T02:49:18ZThere it is. A real X.509 certificate, held by the node’s ztunnel, issued to something called spiffe://cluster.local/ns/bookshop/sa/default, valid from one Tuesday morning to the next.
Two rows, and every field in them is worth a paragraph.
Reading a SPIFFE URI
spiffe://cluster.local/ns/bookshop/sa/default is not a hostname and it is not a DNS name. It is a SPIFFE ID — SPIFFE being the Secure Production Identity Framework for Everyone, an open specification for naming workloads, which Istio implements and which several other systems (SPIRE, various cloud identity brokers) also speak. It goes in the certificate’s Subject Alternative Name field as a URI SAN — a legal, boring, standard place to put it. Any TLS library on earth can read it.
Take it apart:
| segment | value | what it means |
|---|---|---|
| scheme | spiffe:// | this is a workload identity, not a URL you can fetch |
| trust domain | cluster.local | which mesh issued this. Istio defaults it to the cluster’s DNS domain |
/ns/… | bookshop | the Kubernetes namespace |
/sa/… | default | the ServiceAccount |
Read the last row again. The workload’s cryptographic identity — the thing that will decide, in chapter 11, whether a request is allowed or reset — is its ServiceAccount. Not its pod name, which changes on every rollout. Not its IP, which changes faster than that. Not its Deployment, not its labels, not its image. Its ServiceAccount.
The Kubernetes book planted this and then walked away from it. At the end of chapter 11 there, after establishing that every pod runs as somebody whether you asked or not, that the somebody is a ServiceAccount, and that RBAC is a question about that name, there was a paragraph saying the name would come back as a mesh identity. This is it coming back. The identity you configured so that a pod could GET /pods from the API server is now, without you doing anything else, the identity one pod uses to prove to another pod which workload it is. Authorization against the control plane and authentication between your services turn out to be the same name. That is a genuinely elegant piece of design — and it is about to cause us a problem.
The clock: 24 hours, and a ten-year root
Look at the dates on the leaf.
NOT BEFORE 2026-07-14T02:47:49Z
NOT AFTER 2026-07-15T02:49:49ZA hair over 24 hours. That is the entire life of the credential that authenticates every connection catalog makes. (The two minutes of slop at the front are deliberate: the certificate is backdated slightly so that a receiving node whose clock is a little behind does not reject a certificate from the future. Clock skew is the oldest bug in TLS and everyone who ships a CA learns to leave a margin.)
Now the second row:
NOT BEFORE 2026-07-14T02:49:18Z
NOT AFTER 2036-07-11T02:49:18ZThe root is good for ten years — and its NOT BEFORE is the morning I installed Istio. That tells you something important that nobody puts on the front page: by default, istiod is your certificate authority, and it generated a self-signed root for itself at install time. It holds the key in a Secret in istio-system. There was no ceremony, no HSM, no offline key, no external CA — you ran istioctl install and a ten-year trust anchor for your entire production mesh came into existence in about a second.
This is a reasonable default and a bad long-term posture, and you should know which parts are which. It is reasonable because it makes the mesh work out of the box, and because a self-signed root inside one cluster, protecting service-to-service traffic in that cluster, is not obviously worse than the nothing you had before. It is a bad long-term posture because that key is a Kubernetes Secret — and we spent a chapter of the Kubernetes book establishing that a Secret is base64, not encryption — so anyone who can read Secrets in istio-system can mint a certificate for any identity in your mesh and impersonate anything. Also, the moment you want two clusters to trust each other, they need a common root, and a root that istiod invented for itself is not one. Production meshes plug in an intermediate CA signed by a real corporate root, which Istio supports and which I did not do here — treat that as the path to look up rather than a thing I ran.
Why so short a leaf? Because a stolen certificate is only useful while it is valid, and the entire security argument for a mesh rests on that sentence. Compare the alternatives that a certificate replaces. A shared API key lives in a ConfigMap until somebody rotates it, which is to say forever. A database password gets copied into a laptop in 2023 and still works in 2026. A long-lived certificate is the same problem in a nicer format. A 24-hour certificate that ztunnel obtained by presenting the pod’s own projected ServiceAccount token, and that is never written to disk in your pod because your pod does not have it, is a credential that an attacker cannot meaningfully steal and cannot usefully keep.
Be careful about how you state the rotation, though, because this is exactly the sort of number people repeat wrongly. What I measured is the validity window: 24 hours. ztunnel does not wait until the last minute to renew — it refreshes ahead of expiry, so that a certificate is always replaced well before anything can break — but the refresh cadence is not something I timed, and I am not going to quote you a figure I did not watch. The right mental model is: the window is a day, the renewal happens automatically, and the practical consequence is that no human ever touches a certificate in this system. Nobody gets paged at 2 a.m. because a cert expired. That failure mode — which has taken down more production systems than most outages you can name — is simply not available here.
The reckoning: every service had the same identity
Go back to the certificate dump and read the name again, slowly, because I skipped past it and you may have too.
spiffe://cluster.local/ns/bookshop/sa/defaultsa/default.
Now think about which workloads run on k8s-lab-worker. Look at the workload table at the top of this chapter: catalog and web are both up there. And there is one leaf certificate on that node’s ztunnel, and it is issued to default.
Every service in the bookshop was running as the default ServiceAccount. web, catalog, orders, postgres, the shelf controller. All of them. So in SPIFFE terms, all of them had the same identity:
web -> spiffe://cluster.local/ns/bookshop/sa/default
catalog -> spiffe://cluster.local/ns/bookshop/sa/default
orders -> spiffe://cluster.local/ns/bookshop/sa/default
postgres -> spiffe://cluster.local/ns/bookshop/sa/defaultSit with what that means for a second, because it is more damaging than it looks.
The mesh is doing everything it promised. The connections are encrypted. Each one carries a client certificate. The certificate is verified against the root. The identity in it is authentic — genuinely, cryptographically, unforgeably authentic. And the identity is default, for everybody.
So a policy that says “only orders may connect to postgres” is not a policy. It cannot be written, because there is no such identity as orders. The best you can express is “only default may connect to postgres”, and default is web, and default is the shelf controller, and default is any pod anybody deploys into this namespace tomorrow afternoon without thinking about it. The rule would be satisfied by everything, which means it constrains nothing, which means it is not a security control — it is a comment.
Here is the sentence I want you to take out of this chapter:
mTLS without distinct ServiceAccounts is encryption without authorization. You get a perfectly secure channel between two things you cannot tell apart.
And it is worth being clear about whose fault this is, because the interesting answer is nobody’s. The Kubernetes book was not wrong. It said, correctly, that every pod that does not name a ServiceAccount gets default, and it ran the bookshop that way, and that is what almost every real application in the world does. Nothing in Kubernetes punishes you for it. Nothing warns you. The pods run, the app works, the tests pass, and default is a perfectly serviceable identity right up until the moment something asks you to distinguish between your workloads — at which point you discover you have been running a fleet of services that are, as far as the platform is concerned, indistinguishable from one another.
The mesh did not create this problem. It revealed it. It is the first system in your stack that had a reason to ask “who is this, exactly?”, and the answer came back “somebody in the bookshop namespace”, and that is the moment you learn what your identity hygiene has actually been all along. If that lands as an indictment of the mesh — look at the extra work it made me do — you have it exactly backwards. The work was always required for the thing you wanted. You just did not have a system that would tell you.
Fixing it
The fix is four lines per service, and it is the least glamorous YAML in this book.
apiVersion: v1
kind: ServiceAccount
metadata:
name: catalog
namespace: bookshopand, in the Deployment’s pod template:
spec:
template:
spec:
serviceAccountName: catalog
containers:
- name: catalog
...Repeat for orders and web. That is it. The ServiceAccount is, as the Kubernetes book put it, the most boring object in Kubernetes — a name, a namespace, and nothing else. It grants no permissions. It has no fields worth showing. It exists so that things can point at it.
Two consequences follow immediately, and one of them costs you a restart.
It costs a restart. serviceAccountName is part of the pod spec, so changing it changes the pod template, so the Deployment rolls. This is the one thing in this chapter that will bounce your pods — and note the irony precisely, because it is instructive: enrolling in the mesh cost you nothing, and the thing that cost you a rollout was fixing a Kubernetes-level mistake that predates the mesh entirely. Do this before you enrol, if you have the choice. Do it on day one of any new service, whether or not a mesh is anywhere in your plans, because a distinct ServiceAccount with zero permissions costs you nothing and is the prerequisite for every access-control decision you will ever want to make about that workload.
The identity changes immediately after. With serviceAccountName: orders in place, that pod’s SPIFFE ID becomes spiffe://cluster.local/ns/bookshop/sa/orders — and it is now a name that means one thing. The next chapter writes an authorization policy that names exactly that principal, and the policy admits orders and resets the connection from everybody else. That policy is only possible because of the four lines above. This is the payoff, and it arrives one chapter later.
While you are in there: the Kubernetes book also told you to set automountServiceAccountToken: false on workloads that never call the API server, which the bookshop’s services do not. That advice still stands and it does not conflict with anything here. The pod still has a ServiceAccount — it still runs as somebody, and that name is still what ztunnel gets a certificate for — it simply is not handed an API bearer token it has no use for. Identity for the mesh, no credential for the API. That is exactly the combination you want.
What this is not
Three honest limits, because the failure mode of a chapter like this is that you close the tab believing you are finished.
mTLS is between the ztunnels, not between your processes. The encrypted, authenticated session runs from the ztunnel on the source node to the ztunnel on the destination node. The two short hops at the ends — application to local ztunnel, and local ztunnel to application — are inside a single node, unencrypted, over the pod network. For essentially every threat model this is fine — those hops never touch the wire, and an attacker who is already root on your node has much better options than sniffing loopback. But if somebody in a compliance conversation asks you whether traffic is encrypted “end to end”, the honest answer is that it is encrypted node to node with a plaintext handoff inside each node, and you should say so rather than nodding.
It is permissive, and unenrolled clients still get in. We proved this in the last chapter and it is worth repeating here, in the chapter where everything feels solved:
orders (in mesh, SA=orders) -> HTTP 200
intruder (ns outside, no mesh) -> HTTP 200The intruder pod is in another namespace, has no ambient label, has no identity, presents no certificate, and talks plain unencrypted HTTP straight to catalog. It gets a 200. Enabling the mesh did not shut a single door. Istio’s default posture is PERMISSIVE — mesh clients get mTLS, non-mesh clients are still accepted — because the alternative would break every Prometheus scrape and every legacy caller the instant you typed kubectl label, and a mesh you cannot adopt incrementally is a mesh you cannot adopt.
Requiring mTLS is a PeerAuthentication with mtls.mode: STRICT, which is documented and which I did not run in this lab. What I did run is the authorization route — an AuthorizationPolicy that names the principals it will accept — and that is what the next two chapters do, because it is the more useful lever: it does not just require a certificate, it requires a particular identity.
Identity is not authorization. You now know, cryptographically, who is calling. Nothing is checking it. The certificates are being presented, validated, and then essentially ignored — because you have not yet written a single rule that consults them. Everything in this chapter is the plumbing for a decision that has not been made yet.
Final thoughts
The impressive-sounding half of this chapter is the one that took no work at all. Mutual TLS between every service, with automatically rotated 24-hour certificates, issued by a CA you did not configure, with zero code changes and zero restarts, is a thing that used to be a multi-year programme at a large company and is now a kubectl label. It is worth pausing on that. Whatever else you think about service meshes, the transport-security problem — the one that generated a decade of internal PKI projects, key-distribution designs, and sidecar-mounted certificate wranglers — is, for practical purposes, solved and commoditised.
The unimpressive-sounding half is the one that matters. It turns out the hard part of “who is calling this service?” was never the cryptography. It was that nobody had ever been forced to give the callers names. We shipped a working application to a real cluster, gave it a database, exposed it through a gateway, wrote RBAC for it — and every process in it was called default. Not because anyone decided that — because nobody decided anything, and default is what you get when you do not decide.
That is the actual service the mesh performed here, and it is worth more than the encryption. It asked a question the rest of the stack never asks — which one of you is this? — and in answering it we found out that for the entire life of the application, there had been no answer. Every access-control system you will ever build stands on the ability to tell your own workloads apart. It is worth finding out early whether you can.
Comments