ztunnel Cannot Send You a 403

The node proxy that carries every packet in an ambient mesh — what it enforces, why its denials arrive as a connection reset rather than an HTTP status, and the hard ceiling it hits the moment you ask it about a URL.

Here is the output that this chapter exists to explain. Two pods, the same URL, the same second, with a policy in place that permits one of them and not the other:

$ curl http://catalog/books            FROM orders (the permitted principal)
  http=200
  curl exit=0

$ curl http://catalog.bookshop/books   FROM the unenrolled intruder
  http=000
  curl exit=56

The first one is unremarkable. The second one is the whole ambient security model in two lines — and it is not what you expected.

You expected 403. Every tutorial you have read shows a 403. The Istio documentation’s own denial examples show 403, with a body reading RBAC: access denied. Your monitoring dashboards have a panel for 4xx. Your integration test asserts assert resp.status_code == 403.

There is no 403 here. There is no status code at all. http=000 is what curl -w '%{http_code}' prints when there was no HTTP response whatsoever — no status line, no headers, no body, nothing came back. And exit code 56 is curl’s CURLE_RECV_ERROR: “failure with receiving network data” — in practice, connection reset by peer. The TCP connection was accepted, and then it died.

That is not a bug, a misconfiguration, or a rough edge in a young feature. It is the correct and inevitable behaviour of the component that denied the request, and once you understand why, you will understand the entire architecture of ambient mode — what it can do for you cheaply, and what it fundamentally cannot do at all.

The component

Here is the entire ambient data plane, as the install chapter left it — three objects, and that is all of them:

Deployment   istiod           <none>   <none>   1
DaemonSet    istio-cni-node   3        3        <none>
DaemonSet    ztunnel          3        3        <none>

Three nodes, three ztunnels — desired 3, ready 3. It is a DaemonSet — one pod per node, forever, which is exactly what a DaemonSet is for and exactly what it was in the Kubernetes book’s chapter on them. Every pod on a node shares that node’s ztunnel. It is written in Rust, it is not Envoy — and it was built from scratch for this one job rather than adapted to it.

Hold the arithmetic in your head, because it is the entire efficiency argument for ambient and it is not subtle. In the sidecar model, a node running one hundred pods runs one hundred proxies — one hundred copies of Envoy, each with its own memory, its own cached configuration, its own connection pools, its own startup time — and its own CVE. In ambient, that node runs one. Not one per Deployment, not one per team — one per node.

And it is outside your pod. That matters in ways that go past memory. A sidecar shares your pod’s memory limit and your pod’s lifecycle: it competes with your application for the resources you requested, it must be up before your app can talk to anything, and it must be down after your app finishes — which is precisely the ordering problem that made Jobs hang forever for years, because the proxy never exited and the pod never completed. That particular bug is fixed: Kubernetes’ native sidecars closed it, Istio 1.30 uses them by default, and a Job with a sidecar now completes — I run one in the sidecar chapter. What has not changed is the arithmetic: your pod’s memory limit is still shared with an Envoy, and there is still one Envoy per pod. When the proxy is a DaemonSet on the node, your pod’s memory limit is your pod’s alone.

What it does

ztunnel has four jobs, and it is worth listing them precisely, because the list is short and the omissions are the point.

1. mTLS and HBONE. It terminates and originates the tunnels from the last chapter. Every connection into and out of an enrolled pod on this node goes through this process, gets wrapped in a mutually authenticated TLS session to the peer node’s ztunnel, and gets unwrapped at the other end.

2. Workload identity. It holds the SPIFFE certificates. It presents the right one for the right source workload — one ztunnel, many identities — and it must keep them straight, which is why the certificate table in the last chapter was per-node rather than global.

3. L4 authorization. It decides whether a TCP connection may be established at all — based on the identity in the client certificate, the source and destination workloads, the namespace, and the port.

4. L4 telemetry. It counts connections and bytes, and — this is the good part — it labels them with the workload identities at both ends.

That is the list. Read it again and notice that every single item on it stops at the transport layer.

The policy, and what it can express

Here is the policy that produced the output at the top of the chapter. The bookshop’s catalog service holds the book data; the only service that has any business reaching it directly is orders, which calls it to validate an ISBN.

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: catalog-l4
  namespace: bookshop
spec:
  selector:
    matchLabels:
      app: catalog
  action: ALLOW
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/bookshop/sa/orders"]

Everything in that file is a payoff from an earlier chapter.

principals: ["cluster.local/ns/bookshop/sa/orders"] is the SPIFFE ID from the last chapter, minus the spiffe:// scheme, which Istio strips in policy. It is the ServiceAccount, which is the identity, which is the certificate. And it is a name that means one thing only because we gave orders its own ServiceAccount — had we left the bookshop as we found it, this line would have had to say sa/default, and sa/default was every workload in the namespace, and the policy would have permitted the entire shop while looking like it permitted one service. That is not a hypothetical; that is the state the application was in one chapter ago.

action: ALLOW with one rule means what a firewall means: for the selected workloads, permit what matches and deny everything else. An ALLOW policy is a whitelist, and its arrival flips the workload it selects from open to closed. This is worth being loud about, because it is the single most common way people break their own cluster with Istio: the policy above does not only lock out the intruder, it locks out web as well, and anything else that was quietly calling catalog. A real version of this policy names every legitimate caller. In the lab I exercised it with exactly two callers — the one it names, and one it does not — because those are the two cases that teach the mechanism. The version you ship needs a list, and the way you build that list is to look at the telemetry before you write the policy, which is a sentence I will come back to.

The rest of the L4 vocabulary

principals is the one worth learning first, because it is the one that is actually strong — a principal is a cryptographic claim, not an assertion the caller makes about itself. But the L4 rule language has more in it, and all of it is available to ztunnel because all of it is knowable without reading a byte of the payload:

  • namespaces — “anything in bookshop”, which is a coarser and much more common policy than naming individual ServiceAccounts, and a reasonable first step for a team that has not yet done the ServiceAccount work.
  • to.operation.ports — the destination port. Layer 4 knows about ports; that is more or less its definition.
  • ipBlocks / remoteIpBlocks — CIDR matching on the source, for callers that have no mesh identity at all.
  • notPrincipals, notNamespaces — negations, which combine with an action: DENY policy to express “everything except”.

The DENY action deserves a warning that will be familiar if you read the Kubernetes book’s RBAC chapter and noticed that RBAC has no deny rule at all. Istio does have one, and it evaluates before ALLOW — a DENY that matches wins, regardless of what any ALLOW says. That is more expressive than RBAC and correspondingly easier to get wrong, and the standard advice holds: prefer a default-deny posture built out of ALLOW policies, and reach for DENY only for the narrow cases where you genuinely mean “never, no matter what else says otherwise”.

I ran the principals form and nothing else in this lab. The four bullets above are documented behaviour I did not exercise, and I would rather tell you that than let you assume the whole rule language has been through the same test as the one line I actually ran.

And selector: — that is the third payoff, and the subtlety worth carrying into chapter 11.

selector: versus targetRefs:

An AuthorizationPolicy can be attached to things in two different ways, and they are not interchangeable.

selector: picks workloads by their labels. It means “these pods”. A policy bound this way is enforced by the thing that sits in front of those pods at the transport layer — which, in ambient, is the ztunnel on each of their nodes. Workload-scoped. This is the binding L4 policy uses, and it is what the policy above uses.

targetRefs: picks a Kubernetes object — most usefully kind: Service. It means “this service”. A policy bound this way is enforced by whatever is handling that service’s traffic at layer 7, which in ambient is a waypoint, and if there is no waypoint there is nothing to enforce it.

The distinction sounds academic and it is anything but. It is the seam that the fail-open footgun lives in, and chapter 11 will spend most of its length on that. For now, hold one sentence: workload-scoped selector: policies are enforced by ztunnel and are therefore limited to what ztunnel can see; service-scoped targetRefs: policies are enforced by a waypoint and can see everything a waypoint can see. What ztunnel can see is the subject of the rest of this chapter.

Why the denial is a reset

Now the beat this chapter was written for.

orders connects. ztunnel on catalog’s node accepts the HBONE tunnel, reads the client certificate, finds spiffe://cluster.local/ns/bookshop/sa/orders, checks it against the policy, matches — and passes the bytes through. The application answers. http=200, curl exit=0.

The intruder connects. It is not in the mesh, so there is no tunnel and no client certificate — it is a plain TCP connection from an unknown source. ztunnel checks the policy, finds no matching principal, and denies.

And now ask the only question that matters: how, physically, does ztunnel tell the intruder “no”?

It cannot send a 403. Not “it chooses not to” — it cannot. A 403 is an HTTP response. Sending one means composing a status line, writing headers, writing a body, and speaking HTTP back down the socket. To do that, ztunnel would have to be an HTTP server. It is not one — and it has not parsed a single byte of that connection as HTTP, does not know whether the bytes are HTTP at all, and — remember postgres, sitting in the workload table with HBONE next to it — is deliberately built so it does not have to care. It carries TCP. It carries Postgres, gRPC, Redis, and whatever else you invent, with exactly the same code path. A component that speaks every protocol by refusing to speak any of them cannot suddenly produce an HTTP error page.

So it does the only thing available to a transport-layer component that has decided a connection is not permitted: it closes the connection. The socket dies. The client’s recv() fails.

http=000
curl exit=56

000 is the absence of a response. 56 is CURLE_RECV_ERROR. That is what “denied at layer 4” looks like from the client’s chair, and it will look like that in every language and every HTTP library you own — as a connection error, an ECONNRESET, a SocketException, a “socket hang up”, a Connection reset by peer. It will not look like a status code, because there is no status code.

This has a consequence that is worth writing on a card and taping to your monitor:

The shape of a denial tells you which layer denied you. A connection reset means ztunnel. An HTTP 403 means a waypoint. If you are debugging an authorization problem and you know which one you got, you have already halved the search space.

And it has a nastier consequence, which is that a reset is ambiguous with a fault. A connection reset is also what you get from a crashed pod, a full listen backlog, an OOM kill, a node quietly losing its network — the entire vocabulary of ordinary infrastructure failure. The Kubernetes book opened with a rolling update that produced 000s, and the cause there was a race between kube-proxy and a dying container — nothing to do with policy at all. So the failure mode to watch for is a real one: somebody applies an L4 policy, forgets a caller, and the team spends an afternoon debugging what looks exactly like an infrastructure flake. It is not flaky. It is working. It is denying you, in the only vocabulary it has.

Where the decision is made

There is a detail hiding in the intruder’s failure that is worth dragging into the light, because it is the reason this whole model is trustworthy rather than merely tidy.

The intruder is not in the mesh. It has no ztunnel handling its traffic, no redirection rules in its network namespace, no certificate, and no idea that Istio exists. So no component on the intruder’s side could possibly have enforced anything — there was nothing there to enforce it. The check that failed happened at the destination: on catalog’s node, in catalog’s ztunnel, on a connection that had already arrived.

That is not an implementation detail. It is a security property, and it is the one that separates a policy from a suggestion. Any enforcement that happens only on the caller’s side is enforcement you are asking the caller to perform on itself — which is fine when the caller is a well-behaved member of your mesh, and worthless in the exact scenario the policy exists for, which is a caller that is not. The destination must be the one that decides, because the destination is the only party with an interest in the answer. Everything else is advice.

An enrolled caller can and does get checked on its own side too, which saves a pointless round trip across the network for a connection that was never going to be permitted. But that is an optimisation, and the guarantee does not depend on it. Take away the caller’s ztunnel entirely — which is what an unenrolled intruder is — and the policy still holds, because the enforcement was never on that side to begin with.

What ztunnel fundamentally cannot do

Say it in one sentence and then take it apart: ztunnel does not parse HTTP.

It sees a TCP connection between two identities. It sees bytes. It does not see a method, a path, a header, a query string, a body, or a status code — not because it has been configured not to look, but because looking would mean building an HTTP parser into the hot path of every packet on the node, which is exactly the cost ambient was designed to avoid. The decision is architectural and it is deliberate.

Follow that through to its consequences, because each one of them is a feature a service mesh is supposed to have and ztunnel simply does not:

It cannot enforce “GET but not DELETE”. A policy that says methods: ["GET"] requires reading the request line. ztunnel never reads the request line. It can tell you that orders opened a connection to catalog; it cannot tell you what orders then asked for over that connection. In L4 terms, GET /books and DELETE /books are the same thing: some bytes.

It cannot retry a 503. Retrying means noticing that the response was a failure, which means parsing the response, which means HTTP. ztunnel will happily carry your 503 back to you, byte for byte, in perfect confidence that it is doing its job.

It cannot time out a slow request. It can time out a slow connection — TCP has timeouts and ztunnel can act on them — but “this request took more than one second” is a statement about a request/response pair, which is an L7 concept. A connection that sits open for an hour serving fast requests and a connection that sits open for an hour serving one glacial request look identical at the transport layer.

It cannot report a request rate by response code. No parsing, no status codes, no code=500 in your metrics. And no path-level or method-level breakdown either.

It cannot route. Weighted splits, header-based routing, canary deployments by percentage — all of these are decisions made per request, and ztunnel does not have requests. It has connections, and a connection is not a place you can make a routing decision.

Everything on that list requires a full L7 proxy in the path, and in ambient that is a waypoint, which is the next chapter. This is the actual architecture of ambient mode, and it is worth stating as a design rather than a limitation: the cheap thing runs everywhere, and the expensive thing runs only where you ask for it. Every node pays for mTLS, identity, and L4 policy, because those are cheap and everybody wants them. Only the services that need HTTP semantics get an Envoy, and you opt in per service, and you pay for it per service. The sidecar model, by contrast, put a full L7 proxy next to every pod whether that pod needed one or not — which is a fine trade if every service needs L7 features, and a wasteful one if, as in most real fleets, the honest answer is that half of them just want the traffic encrypted.

What you do get: L4 telemetry, with names

It would be easy to read the last section as a list of things ztunnel is bad at. Here is what it is good at, and it is more than people expect.

ztunnel emits L4 metrics for every connection it handles, and — the part that makes them useful rather than merely present — it labels them with the workload identity at each end. After a session of experiments on the bookshop, Prometheus had this, with no application instrumented:

istio_tcp_connections_opened_total
  shelf-controller   462
  orders             191
  waypoint            57
  web                 16
  catalog             11

Nobody added a metrics library. Nobody wrote a line of instrumentation — nobody was asked to. Those numbers arrived because every connection in the namespace passes through a component that knows who both ends are, and it counted them.

Look at the third row before you read on: waypoint 57. That table was taken from a namespace that had a waypoint in it by the time I ran the query, and I am leaving the row in rather than cropping the exhibit to fit the sentence. What ztunnel is counting there is real — the waypoint is just another workload on the node, opening connections like anything else, and ztunnel counts it like anything else. The claim this table supports is that ztunnel gives you identity-labelled L4 telemetry for everything it handles, waypoint or no waypoint. It does not need a waypoint to do it. It would look the same, minus that row, on a mesh that had none. But I did not run that query on a waypoint-free namespace, so I am showing you the table I actually have.

This is exactly the input you need for the thing I promised to come back to: building the caller list before you write an ALLOW policy. The correct order of operations is to enrol, wait, look at who is actually connecting to catalog — including the batch job nobody remembers and the debug pod somebody left running — and only then write a policy that names them. Ambient makes that order of operations natural, because enrollment is a label and it gives you the telemetry immediately, without asking you to commit to anything. Chapter 12 goes properly into what the mesh will tell you for free.

The other thing worth knowing here is what the metrics cannot say, and by now you can predict it: no methods, no paths, no status codes. istio_tcp_connections_opened_total and its byte counters, with identities. That is the full L4 vocabulary.

The blast radius, honestly

A shared, node-level component that carries every packet for every pod on the node is a failure domain, and it would be dishonest to reach the end of a chapter about ztunnel without saying so out loud.

If ztunnel on a node crashes, restarts, deadlocks, leaks, or ships a bad release, every enrolled pod on that node is affected at once. Not one team’s service — all of them. That is a genuinely worse blast radius than a sidecar, where a broken proxy takes down exactly one pod, and where a bad proxy rollout can be canaried one Deployment at a time. This is a real trade and you should price it.

Now price the other side of it. There is one ztunnel per node instead of one proxy per pod, so there is one thing to patch, one thing to watch, one thing to upgrade, and one thing to reason about — where the sidecar model gives you a proxy version embedded in every pod spec in the cluster, a fleet-wide restart to upgrade them, and the permanent possibility that some Deployment nobody has touched in eighteen months is still running last year’s Envoy with last year’s CVE. “One shared component per node” and “one hundred unmanaged components per node” are both risks. They are not the same size of risk, and the second one is much easier to ignore, which is not the same as it being smaller.

And ztunnel is not in your application’s memory space. It cannot be reached by a bug in your app, and it cannot OOM your container. It is a process on the node — run by the platform team, on its own lifecycle, patched on its own schedule. The sidecar’s failure domain was smaller but it was inside your pod, which is why sidecar failures were so consistently reported as application failures.

Final thoughts

The important thing about curl exit=56 is not the number. It is what the number implies about the shape of the system, and it is a lesson that generalises well past Istio.

ztunnel denies you in the only language it speaks. It is a transport-layer component, so it can accept a connection or it can kill one, and that is the entire expressive range of its refusals. It cannot explain itself, because explaining yourself requires a protocol, and having a protocol is exactly the thing it gave up in order to be cheap enough to run on every node and general enough to carry your Postgres traffic. The absence of a 403 is not a missing feature. It is the receipt for a design decision, and you got something good in exchange for it.

Which sets up the question the next chapter answers. You now have a mesh where every connection is encrypted, every workload has a name, and you can decide who may connect to whom — and that genuinely is most of what people actually want from a mesh, delivered for the price of a label and a DaemonSet. But you cannot say “GET but not DELETE”, you cannot retry a failure, and you cannot time out a slow request. Those things need something that reads HTTP.

The honest framing is that the L4 layer is not a compromise on the way to the L7 layer. It is a complete, useful product on its own, and a great many services should stop right here and never deploy anything else. The next chapter is about the ones that should not — and about the small, service-scoped, deliberately opt-in Envoy that gives them what ztunnel cannot. It also comes with a display bug that is not a bug, which will convince you it is broken.

Next: Your Waypoint Says None, and It Is Working Perfectly

Comments