The Policy Said ALLOW. The DELETE Went Through.

The same AuthorizationPolicy, the same caller, and the only variable is whether a waypoint is in the path — because an L7 rule with nothing to enforce it is not a stricter policy, it is an unenforced one, and it fails open.

Here is a policy. It says that orders may read the catalog and may not modify it — GET yes, DELETE and POST no. It is eleven lines of YAML, it is the most ordinary thing in the world, and it applies without complaint:

$ kubectl apply -f catalog-l7.yaml
authorizationpolicy.security.istio.io/catalog-l7 created

$ kubectl get authorizationpolicy -n bookshop
NAME         ACTION   AGE
catalog-l7   ALLOW    29s

Created. ALLOW. Twenty-nine seconds old. Now exec into the orders pod, which is the caller the policy names, and do the exact thing the policy forbids:

GET    /books  -> http=200
DELETE /books  -> http=200
POST   /books  -> http=200

Two hundred. Two hundred. Two hundred.

The DELETE went through. The POST went through. The policy did not deny them, did not log anything about them, and did not fail — because the policy is not running. Nothing in the cluster is enforcing it, nothing will ever enforce it, and the command you would naturally type to check on it prints a row that says ALLOW and looks entirely healthy.

This chapter is the reason this book leads with ambient rather than with sidecars — and the reason a chapter on authorization is the eleventh of twelve rather than a paragraph in the third. The security model in ambient mode is genuinely, materially different from the one nearly every Istio tutorial on the internet describes, and the two ways it differs are both the kind that fail silently. You will not get an error. You will get one of two things: a green cluster with an unenforced policy in it, or a denial that arrives in a form you were not watching for and will not recognise.

The central fact, as a controlled experiment

Everything in this chapter reduces to one table. Same AuthorizationPolicy — literally the same object, never edited. Same caller, orders, authenticated by its ServiceAccount. Same three requests. The only variable in the entire experiment is whether a waypoint is in the path:

requestno waypointwaypoint deployed
GET /books200200
DELETE /books200 — silently unenforced403
POST /books200 — silently unenforced403

Read the middle column again, because it is the whole problem. That is not a policy being permissive — permissive is a decision. That is a policy with no enforcer at all, and Istio ran your cluster happily for as long as you cared to leave it that way.

To understand why, you have to stop thinking of the mesh as one thing that enforces policy, and start thinking of it as two proxies with very different eyesight — which is exactly what chapter six and chapter seven were building toward. ztunnel is on every node, sees every packet, and speaks L4. The waypoint is optional, per-Service, and speaks L7. What a policy can enforce depends entirely on which of the two is asked to enforce it — and that depends on how you attached it.

What ztunnel can actually enforce: identity

Start with the layer you get for free, because you already have it and it is doing more than you think.

Every pod in the mesh has a SPIFFE identity, derived from its ServiceAccount, delivered as a 24-hour X.509 leaf, and presented on every connection over mTLS. That was chapter five, and it is where the ServiceAccounts chapter of the Kubernetes series pays off — the identity you gave the pod so that it could talk to the API server is the same identity the mesh now uses to decide whether it may talk to your services. That is a strange and rather beautiful reuse, and it is why the bookshop had to be fixed before any of this could work at all: every service originally ran as the default ServiceAccount, which meant every service had the same SPIFFE identity, which meant an identity-based policy could not tell web from orders from a compromised debug pod.

With each service on its own ServiceAccount, ztunnel can enforce this:

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: catalog-l4
  namespace: bookshop
spec:
  selector:
    matchLabels:
      app: catalog
  action: ALLOW
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/bookshop/sa/orders"]

Only the orders ServiceAccount may open a connection to the catalog. No methods, no paths, no headers — an identity, and nothing else. That is the entire vocabulary available at L4, and it is a bigger vocabulary than it looks, because “who” is the question most breaches actually turn on.

Note the two fields that make this an L4 policy, because they are the ones that decide who enforces it. It uses selector: — which binds the policy to workloads, the pods matching app: catalog. And its rule conditions on nothing but principals. Both halves matter. A policy bound to workloads is enforced by the thing that sits in front of workloads, in ambient that thing is ztunnel, and ztunnel cannot read an HTTP method.

Apply it, and try it from both sides. From orders, the permitted principal:

$ curl http://catalog/books   FROM orders (allowed principal)
  http=200
  curl exit=0

And from an intruder — a pod in a different namespace, not enrolled in the mesh, with no mesh identity at all:

$ curl http://catalog.bookshop/books   FROM the unenrolled intruder
  http=000
  curl exit=56

An L4 denial is a connection reset. Say it out loud.

We met this in chapter six and I am putting it back on the table anyway, because it is one half of the controlled experiment and because it is the single most important output in the book. Stop and look at what came back.

http=000. Not 403. Not 401. 000 is what curl -w '%{http_code}' prints when there was no HTTP response at all — no status line, no headers, no body. The same 000 the rolling-update chapter produced when a request was routed to a pod whose socket had just closed. There is nothing to parse, because nothing was said.

curl exit=56. From curl’s manual: recv failure: connection reset by peer. The TCP connection was established and then destroyed. That — a dead socket and a puzzled client — is the mesh saying no.

And the reason is not a design quirk, it is a definition. ztunnel is not an Envoy and does not speak HTTP. It is an L4 proxy — it terminates mTLS, it tunnels the connection over HBONE, and it forwards bytes. To return a 403, something must compose a 403, which means writing an HTTP status line onto a stream, which means being an HTTP server — which ztunnel is not. It has exactly one way of expressing a denial available to it, and it is the way every L4 firewall in history has expressed one: it kills the connection.

Now, the sentence I want tattooed somewhere:

An L4 denial is a connection reset. An L7 denial is an HTTP 403.

Nearly every Istio tutorial you will find shows you a 403 for everything — and that is sidecar behaviour. A sidecar is an Envoy. An Envoy is in the path of every request to every pod in the mesh, whether the policy needs L7 or not, and an Envoy speaks HTTP, so even a pure identity denial gets rendered as a nice, legible RBAC: access denied with a 403 status. That is a real property of the sidecar data plane, it was true for years, it is what the docs and the blog posts and the conference demos all show, and it is not what happens in ambient.

So picture the reader who followed one of those tutorials into an ambient mesh. They write an identity policy, they exec into a pod that should be denied, they run curl, and they wait for the 403 they were promised. What they get is an error and an exit code they have never looked up — so they conclude the policy is broken, and they start deleting things.

The policy was working perfectly. It denied the request in the most total way a request can be denied — there was no request, because there was no connection.

And there is a third shape, which I found by auditing this book

The tattoo above has two shapes on it. There are three, and I only found the third by going back and re-running my own claims months after I wrote them.

An L4 denial is a connection reset when the caller reaches the destination’s ztunnel directly. But once a waypoint is bound to the Service — which is the whole subject of the waypoint chapter — an in-mesh caller does not reach that ztunnel directly. The waypoint does, on the caller’s behalf. So when the destination’s ztunnel resets that connection, the party holding the dead socket is the waypoint, not your application. And the waypoint, being an Envoy, does what any proxy does when its upstream hangs up on it: it composes an HTTP response and tells the caller the upstream failed.

caller bypasses the waypoint  ->  L4 reset   ->  http=000, curl exit 56
caller goes through a waypoint ->  L4 reset  ->  the waypoint reports it as  503
caller goes through a waypoint ->  L7 rule   ->  403

Three shapes, and the middle one is the one nobody warns you about, because it looks like a capacity problem. A 503 from a mesh service means “the thing in front of your backend could not talk to your backend” — and one of the reasons it could not is that you told it not to. If you are staring at 503s and your pods are all healthy, do not go looking at replica counts. Go and read your L4 policies.

This has consequences past the confusion, and they are worth thinking through before you go to production:

  • Your monitoring will not see a 403. A denied L4 connection produces no HTTP status code anywhere, so a dashboard built on response codes will show you nothing. What it shows up as is a connection failure at the client, and a connection ztunnel opened and closed at the mesh. You have to know to look at TCP metrics rather than request metrics — which is the next chapter, and it is one of the reasons that chapter matters more than it sounds like it should.
  • A retry policy may retry it. Look back at the retries we just configured. We used retryOn: 5xx, which will not touch a connection reset. But Envoy’s retry vocabulary includes connect-failure and reset, and a caller with a generously-configured retry policy will interpret a policy denial as a transient network fault and hammer it. I did not run that combination, so treat it as an inference rather than a measurement — but it follows directly from the two mechanisms, and it is the sort of thing worth checking in your own mesh before it checks itself in production.

The fail-open footgun

Now back to the policy that opened the chapter, which is the second half of the ambient security model and the more dangerous half.

We want a rule with a verb in it: orders may GET the catalog, and may not POST or DELETE it. That is not an identity question — orders is a permitted identity in both cases. It is a question about what is inside the HTTP request. So the policy has to talk about methods:

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: catalog-l7
  namespace: bookshop
spec:
  targetRefs:
    - group: ""
      kind: Service
      name: catalog
  action: ALLOW
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/bookshop/sa/orders"]
      to:
        - operation:
            methods: ["GET"]
            paths: ["/books"]

Two things changed from the L4 policy, and they are the same two things in a different order.

The binding changed. This policy uses targetRefs, pointing at a Service — not selector:, pointing at workloads. That is the modern ambient binding, and it is not cosmetic. It is a statement about where in the request path the policy lives — attached to the Service, and therefore enforced by whatever L7 proxy is bound to that Service. It is the same idea as the HTTPRoute-with-a-Service-parentRef from the last chapter, and Istio treats it the same way.

The rule changed. It now has a to: clause with operation.methods — an L7 condition. Something in the path has to open the HTTP request and read the method line.

There is no waypoint deployed. Apply it:

authorizationpolicy.security.istio.io/catalog-l7 created
^ Accepted. No error. No warning. Istio is perfectly happy.

And then, from orders:

GET    /books  -> http=200
DELETE /books  -> http=200
POST   /books  -> http=200

An L7 rule with no waypoint in the path is not a stricter policy. It is an unenforced one.

That is the footgun, and it is worth being precise about the mechanism rather than waving at it — the precision is what makes it fixable. The policy is bound to Service/catalog. A Service-bound authorization policy is enforced by the L7 proxy bound to that Service. There is no L7 proxy bound to that Service. So there is nothing that has been asked to enforce it — not ztunnel, which was never handed the policy and could not read a method if it had been, and not a waypoint, because there isn’t one. The policy has no enforcer. It sits in etcd, correctly formed, valid, and matched by nothing.

Notice how much worse this is than a policy that is merely too permissive. A permissive policy is a decision you made — a bad one, perhaps, but yours. This is a decision you believe you made: sitting in Git, reviewed by a colleague, referenced in a compliance document, and doing absolutely nothing. And it fails open — the direction that costs you a breach rather than an outage, and the direction a security system is meant to be designed never to fail in.

The correction: it does leave evidence, in two places you have to already know about

When I planned this series I wrote down, confidently, that the fail-open “does not error, it does not warn, it just does not apply.” Then I ran it — and that sentence is half true, and the half that is wrong is the useful half. Istio 1.30 does report it. It reports it in two places, and in neither of them will you trip over it by accident.

First, the policy’s own status. The object knows:

$ kubectl get authorizationpolicy catalog-l7 -n bookshop -o jsonpath='{.status}'
  {
      "conditions": [
          {
              "lastTransitionTime": "2026-07-14T02:52:07.824582232Z",
              "message": "Service bookshop/catalog is not bound to a waypoint",
              "observedGeneration": "1",
              "reason": "AncestorNotBound",
              "status": "False",
              "type": "WaypointAccepted"
          }
      ]
  }

WaypointAccepted: False. reason: AncestorNotBound. And a message in plain English that tells you exactly what is wrong and exactly how to fix it: Service bookshop/catalog is not bound to a waypoint. Istio knows. Istio wrote it down.

Second, istioctl analyze:

$ istioctl analyze -n bookshop
Warning [IST0171] (AuthorizationPolicy bookshop/catalog-l7) A condition with a negative status is
present: type=WaypointAccepted, reason=AncestorNotBound, message=Service bookshop/catalog is not
bound to a waypoint.
Info [IST0118] (Service bookshop/postgres) Port name pg (port: 5432, targetPort: pg) doesn't follow
the naming convention of Istio port.

There it is, IST0171, as a warning, with a code you can grep for. (And a free bonus finding from our own manifests — IST0118, the bookshop’s Postgres Service names its port pg, and Istio would like tcp- or http- prefixed port names so it can infer the protocol. That is a real defect in a manifest I wrote in the previous series, and analyze found it in the same breath.)

And now what it does not do, which is the whole reason the footgun exists:

$ kubectl get authorizationpolicy -n bookshop
NAME         ACTION   AGE
catalog-l7   ALLOW    29s

NAME, ACTION, AGE. That is the entire printed output of the command you will actually type. No STATUS column. No READY column. No asterisk, no False, no hint of any kind that the object in front of you is inert. A policy that is enforcing perfectly and a policy that is enforcing nothing print identical rows.

That is not a bug either — it is a consequence of something we built in the CRDs and controllers chapter. kubectl get on a custom resource prints exactly the columns the CRD’s author declared in additionalPrinterColumns, and no more. For AuthorizationPolicy those columns are the name, the action, and the age — so the health of the object is not among them, and it is therefore invisible to the command everyone types, forever, by construction.

So the accurate teaching — better than the one I had written down, and true — is:

It enforces nothing. kubectl get looks fine. And the warning is in two places you have to already know to look.

Which gives you two concrete things to do, and you should do both:

  1. Check .status.conditions on your AuthorizationPolicies. Not once, at the moment you write them — in an alert, on a schedule, the way you would check any other control that is supposed to be running right now.
  2. Run istioctl analyze in CI. It exits non-zero on warnings, it is fast, it needs nothing but read access, and it catches this entire class of bug — the well-formed object that lands nowhere — which is precisely the class you cannot catch by reading the YAML, because the YAML is fine. If you take one operational habit from this chapter, take that one.

Add the waypoint, and the 403 arrives

Nothing about the policy is going to change. It stays exactly where it is, byte for byte. We are going to change one thing about the cluster: put an L7 proxy in the path.

$ istioctl waypoint apply -n bookshop --enroll-namespace --wait
  ✅ waypoint bookshop/waypoint applied
  ✅ waypoint bookshop/waypoint is ready!
  ✅ namespace bookshop labeled with "istio.io/use-waypoint: waypoint"

And the same three requests, from the same pod, against the same policy:

GET    /books  -> http=200
DELETE /books  -> http=403
POST   /books  -> http=403

403. 403. A real HTTP status, in a real HTTP response, that a client library will surface as an error you can catch and a dashboard will count as a 4xx.

Same policy. Same caller. Same request. The only thing that changed is that something in the path can now read a method — a waypoint, which is an Envoy, which speaks HTTP, which can therefore open the request, compare DELETE against methods: ["GET"], find it does not match, and compose a refusal in a language the caller understands.

The policy’s status condition flips with it:

WaypointAccepted=True (Accepted)

AncestorNotBound is gone. The ancestor is bound. The policy has an enforcer.

One last thing, and it will confuse you if nobody says it — because it confused us in chapter seven. With that working waypoint in place, enforcing 403s as you watch, the per-pod view still says this:

NAMESPACE          POD NAME                       ADDRESS     NODE               WAYPOINT PROTOCOL
bookshop           catalog-7c74bb4c8c-hc22b       10.244.2.51 k8s-lab-worker2    None     HBONE
bookshop           catalog-7c74bb4c8c-v649p       10.244.1.64 k8s-lab-worker     None     HBONE

WAYPOINT: None. That is correct. Waypoints are bound to Services, not to workloads, and the per-Service view is the one that tells the truth:

NAMESPACE    SERVICE NAME    SERVICE VIP     WAYPOINT ENDPOINTS
bookshop     catalog         10.96.228.164   waypoint 2/2
bookshop     orders          10.96.2.2       waypoint 1/1

Two views, two answers, both right. Read the one that matches how the thing is actually bound.

ALLOW, DENY, and the flip that catches everyone

Now the semantics, and I am going to be careful here about which parts I watched and which parts I am relaying, because this is precisely the area where a confident half-truth gets somebody breached.

What I ran was ALLOW policies. Here is what they do, and it is the rule people get wrong.

A workload with no AuthorizationPolicy selecting it allows everything. That is the state your mesh is in the moment you enrol it, and it is why enrolment in chapter four did not break the shop. mTLS was suddenly everywhere and every single request still worked, because encryption is not authorization and the mesh had not been told to refuse anything.

The moment one ALLOW policy selects that workload, the workload becomes deny-by-default. Not “the policy adds a permission on top of the existing free-for-all” — the policy replaces the free-for-all. From then on, a request is allowed if and only if it matches at least one rule in at least one ALLOW policy that applies. Everything else is denied.

That single sentence is the most common way people take down their own service with Istio. You write an ALLOW policy to let orders into the catalog. It works — orders gets in. And web, which also calls the catalog, and which you forgot about entirely, is now being refused, because it is not named in the only ALLOW policy in play. You did not write a rule denying web. You wrote a rule that failed to allow it — which in a deny-by-default world is the same thing, arrived at by a different route.

From the documentation, and not from my lab, since I ran only ALLOW policies:

  • Istio evaluates CUSTOM first, then DENY, then ALLOW. A DENY that matches wins over an ALLOW that also matches. DENY is the tool for a blanket exclusion — no one may reach /admin, from anywhere, ever — and it is evaluated before the allow-list gets a say.
  • An ALLOW policy with an empty rules list matches nothing, and therefore denies everything for the workloads it selects. That is not a hypothetical footgun; it is the shape you get by accidentally deleting a block of YAML. I did not run it. Do not find out the hard way.
  • A policy in the root namespace (istio-system, by default) applies mesh-wide rather than to one namespace. Also not run here — but it is how a platform team ships a baseline everybody inherits, and it is the object you should be most careful about reviewing.

If you have a way to run AuthorizationPolicy in dry-run mode — Istio documents an istio.io/dry-run: "true" annotation, which evaluates the policy and reports what it would have denied without denying it — that is obviously the right way to introduce your first restrictive policy into a live system. I did not exercise it in this lab, so I am pointing at it rather than demonstrating it, and you should confirm it behaves as advertised in ambient before you rely on it.

Getting to zero trust without taking down the shop

The end state everybody wants is “every service explicitly declares who may call it, and nothing else gets in”. The way to get there is not to write that policy. The way to get there is to arrive at it in an order where every step is reversible.

Start with the call graph you actually have, not the one on the architecture diagram. You cannot write an allow-list for callers you do not know about, and the deny-by-default flip means every caller you miss is an outage. The mesh already knows the answer — the telemetry in the next chapter hands you the real, observed, source-to-destination graph with workload identities attached, for free, without instrumenting a thing. Write your policies from that, reconcile it with the diagram afterwards, and be prepared for the diagram to be the one that is wrong.

Do identity before verbs. An L4, selector:-bound, principals-only policy costs nothing — no waypoint, no extra proxy, no L7 parse on the hot path — and it is enforced by ztunnel on every node you already have. It gets you the majority of the value: only these named services may open a connection to this one, and everything else on the network is refused at the transport layer. Do that first, everywhere, for every service. It is cheap and it is real.

Then buy L7 where the verbs matter. Not everywhere. A waypoint is a real pod with real memory and a real hop in the request path, and the honest thing to say is that most of your services do not need one. The ones that do are the ones where “who” is not a fine enough question — where the same caller is allowed to read and not allowed to write, where a path prefix is dangerous, where you need to distinguish GET /books from DELETE /books. Deploy the waypoint for those Services, and then write the L7 policy, in that order, so you never spend a minute in the state at the top of this chapter.

And check that it landed. Every time. .status.conditions, istioctl analyze, in CI, on a schedule. A security control you have not verified is running is a security control you do not have.

Final thoughts

The worst kind of security control is not a weak one. A weak control is at least honest about what it is — you can look at it, reason about it, and decide it is not enough. The worst kind is the control that is absent while appearing present, because it does not merely fail to protect you, it consumes the attention you would otherwise have spent on protecting yourself. Somebody wrote the policy. Somebody reviewed it. It is in the repository, it is in the runbook, and its name is in a compliance spreadsheet with a green tick beside it. And it is not running, and nothing anywhere in the system is going to raise its hand.

Ambient makes that failure possible in a way sidecars did not, and I want to be straight about that rather than sell around it. With a sidecar, every pod had an Envoy whether you wanted one or not — you paid for a full L7 proxy on every workload in the mesh, in memory, in latency, in restarts, forever, and in exchange the policy always had an enforcer standing right there. It could not fail to land. Ambient unbundles that. It says: here is L4 and identity, on every node, for almost nothing — and if you want L7, name the Service and pay for a waypoint. That is a much better deal, and I would take it every time. But it is a deal with a seam in it, and the seam is exactly where a policy can fall through.

So learn the two shapes of no, and learn them properly, because they are the two shapes of your entire security posture. The connection that dies with no answer is ztunnel refusing an identity — it is working, it is total, and it will never show up in your HTTP dashboards. The 403 is a waypoint refusing a request — it is legible, it is countable, and it only exists because you deployed something to make it exist. And the request that succeeds when you were sure it would not is the mesh telling you that the thing you wrote is not the thing that is running.

Which raises an awkward question, given everything from the last two chapters. The retries hid a service that was failing half its requests. The ejection hid a pod that was broken. The policy just refused two requests. Every one of those events happened inside a proxy you did not write, to an application that has no idea any of it took place — and none of them made a sound in any log the bookshop keeps.

So how do you see any of it? It turns out you already can, and you have been able to for eleven chapters.

Next: Nobody Instrumented the Bookshop

Comments