The Mesh Doesn't Know What an Order Is

Everything in this chapter is described, not run. I did not stand up an external authorization service, I did not configure the provider, and I did not exercise a single request through it. There is no output below because I have no output to show you, and I am not going to manufacture a plausible 403 from an authorization service that never existed.

That is an unusual way to open a chapter and I want to say why it is here rather than dropped. The previous eight chapters were built on a real cluster, and the three biggest findings in this book — the egress regression, the JWT footgun, the unenforced L7 policy — all came from running the thing rather than reading about it. That is precisely why I am not going to quietly switch registers now and present documentation as experience. ext_authz is important, it is the answer to the question chapter eight ended on, and you need to understand it before you build anything serious. So: an honest map, clearly labelled as a map, with the parts I would test first marked in red.

Read this chapter for the shape of the thing and the decisions it forces on you. Do not read any of it as verified behaviour on Istio 1.30.2.

The question chapter eight could not answer

The last chapter got you to a strong place. The mesh verifies the JWT’s signature against your identity provider’s keys. It requires that a token is present, so anonymous requests never reach your service. It can match a coarse claim — a group, a role, a scope — against the method and path.

And then it walks into a wall, and the wall does not move:

PATCH /orders/8821, with a valid token for sub: user-42. May this user edit this order?

The mesh cannot answer that. Not because Istio is missing a feature, but because the answer is not in the request. Order 8821 belongs to somebody, and who it belongs to is a row in your database. No header carries it. No claim encodes it. To decide, something must know your domain — must know that an order has an owner, that ownership is what confers the right to edit, that a support agent may edit anyone’s order but only within thirty days, and that a refund above five hundred pounds needs a second approver.

That is business logic. It is yours, it changes when the business changes, and no proxy is ever going to infer it.

So the industry’s usual answer is the obvious one — put it in the application. Import the authorization library, sprinkle the checks through the handlers, and get on with your life. That works, and for a small system it is the right call. It also has three well-known costs that get sharper as you scale, and the third is the one that eventually forces the issue:

You write it once per language. Your Go services and your Kotlin services and the Python thing nobody owns each need an implementation, and they will drift. Not maybe — will.

It is invisible. The authorization rules are distributed across a thousand call sites in a dozen repositories. Nobody can answer “who can refund an order?” without grepping. A security review of that system is an archaeology project.

You cannot verify it is there. A missing if in one handler is a hole, and a missing if looks exactly like code that was never written, which is to say it looks like nothing at all. There is no manifest of enforcement points to audit. This is a strictly worse version of the problem this book keeps circling — an absent control that leaves no trace — and it is why moving authorization out of the code has been an ambition for as long as there have been microservices.

ext_authz is Istio’s answer, and it is a good one. The mesh calls out to a service you wrote, hands it the request metadata, and enforces the allow or deny it gets back. Your business logic ends up in the request path — for every service, in every language, at a chokepoint you can see and audit — without a library in any of them.

How it is wired

Two objects — and they live in different places, which trips people up.

First, a provider, in the mesh config. This is a mesh-wide declaration that a particular external service exists and can be called for authorization decisions. It goes in meshConfig.extensionProviders — the same meshConfig that held outboundTrafficPolicy in chapter seven, which is to say the same file whose settings you should now be in the habit of asking which data plane actually reads this.

meshConfig:
  extensionProviders:
    - name: bookshop-authz
      envoyExtAuthzHttp:
        service: authz.bookshop.svc.cluster.local
        port: 8000
        includeRequestHeadersInCheck: ["authorization", "x-request-id"]

There are two flavours. envoyExtAuthzHttp calls your service over plain HTTP: the proxy makes a request to it, and reads the response status to decide. envoyExtAuthzGrpc calls it over gRPC, using Envoy’s defined authorization protocol, which gives you a typed request and a typed response rather than a status code and some headers. The gRPC interface is richer and is what most serious implementations use. The HTTP one is easier to write in an afternoon, and much easier to debug with curl.

Note includeRequestHeadersInCheck. The proxy does not send the whole request to your authorizer by default — it sends metadata, and you list what else you want. That default is doing you a favour: your authorizer probably does not need the request body, and sending it would put your entire payload through a second network hop on every single call.

Second, a policy, with the one action nobody uses.

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: orders-ext-authz
  namespace: bookshop
spec:
  targetRefs:
    - group: ""
      kind: Service
      name: orders
  action: CUSTOM
  provider:
    name: bookshop-authz
  rules:
    - to:
        - operation:
            paths: ["/orders/*"]

action: CUSTOM. This is worth stopping on. The foundations book named it once, in passing, when it listed the evaluation order — and then never used it, because there was nothing to point it at. This is the only thing it is for. AuthorizationPolicy has four actions — ALLOW, DENY, AUDIT, and CUSTOM — and CUSTOM exists for exactly this. It is not “a custom rule”. It means: do not evaluate this yourself, ask the named provider. The provider.name must match a provider declared in meshConfig.extensionProviders, and if it does not, you have a policy pointing at nothing — a possibility I would put near the top of the list of things to test, given everything else in this book.

The rules block still matters, and its role changes. It is no longer deciding anything. It is deciding what gets asked. Requests matching those rules are sent to the authorizer; requests that do not match skip it entirely and fall through to the rest of the policy evaluation. That is your scoping tool, and it is also your latency budget — every path you list here is a path that now costs an extra network round trip.

What your service gets, and what it must say

Your authorizer is an ordinary service — it runs in the cluster, it is in the mesh, and it has a Deployment and a Service like everything else. It is also, from the moment you wire it up, the most important service you own.

On each check it receives the request’s metadata — method, path, host, and the headers you asked for — along with the source and destination of the call. Crucially, if you also have a RequestAuthentication in play, the mesh has already validated the JWT by the time your authorizer is consulted, which means your service can trust the token rather than re-verifying it. That composition is the point: the mesh does the cryptography, and you do the domain.

It answers with one of two things. Allow — for the HTTP provider, a 2xx; the request proceeds, and you may attach headers to it that get merged into the upstream request, which is how you pass a decision, a tenant ID, or a role down to the application. Deny — a 403, or whatever you configure; the request stops at the proxy and never reaches your service.

That header-injection path is more useful than it first appears. Your authorizer has just done the work of figuring out who the caller is and what they may do. Rather than throwing that away and making the application do it again, hand it down: x-user-id, x-tenant, x-permissions. The application trusts them because it can only be reached through the proxy that set them — which is a sentence you should read twice, because it is only true if that is genuinely the case, and a pod that can be reached directly, bypassing its waypoint, can be handed any header an attacker likes. This is exactly where an L4 identity policy from the foundations book earns its place: lock the workload down so the only thing that may open a connection to it is the proxy in front of it.

The decision that matters more than all the others

What happens when your authorizer is down?

Sit with that one, because it is the single most consequential thing in this chapter, and it is a configuration choice you will make in about four seconds while thinking about something else.

Fail closed. The authorizer is unreachable, so every request is denied. Your authorization control holds under failure, which is what a security control is supposed to do — and your entire application is down, because a single service you wrote, probably in a hurry, is now a hard dependency on the request path of every request in the mesh. You have built a beautiful, correct, mesh-wide single point of failure.

Fail open. The authorizer is unreachable, so every request is allowed. Your application stays up, serenely, while the control that decides who may refund an order is not running. And — you have read seven chapters of this book, you know what comes next — nothing tells you. Requests succeed. Dashboards are green. The one signal that something is wrong is a metric on a proxy that nobody has an alert on. That is the egress regression again, wearing a different hat: an unenforced control, a healthy-looking cluster, and a period of time whose length you will only be able to establish afterwards, from logs, under duress.

Neither option is good. That is not a failure of imagination on Istio’s part, it is the actual structure of the problem: you have made authorization a synchronous network call, and networks fail, and there is no third answer to what a failed network call means.

What you can do is stop treating it as a switch and start treating it as an engineering problem:

• Make the authorizer boringly available. Multiple replicas, spread across nodes, no shared database on the hot path if you can avoid it, and a latency budget it is expected to hold. It is now tier-zero infrastructure. Staff it accordingly — this is not a side project for a quiet Friday.
• Cache decisions. The same user asking the same question about the same resource, seconds apart, does not need a fresh round trip. A short TTL turns a hard dependency into a soft one for the duration of a blip.
• Alert on the failure mode itself. If you fail open, then “authorizer unreachable” is a page, not a warning, and it is a page precisely because nothing else will break. The whole danger of fail-open is that it is silent, so you have to manufacture the noise yourself.
• Consider the sidecar-of-the-authorizer pattern. The common production shape is to run the policy engine next to the thing it protects — an OPA sidecar, or a local agent — so that “the network is down” and “authorization is down” stop being the same event. It costs you memory per pod, which after chapter eleven’s numbers you now know how to price honestly.

I did not run any of this. I am telling you what the shape of the problem is, and I am telling you that the four seconds you spend on the fail-open flag will matter more than the four weeks you spend on the policy language.

Evaluation order, and where CUSTOM sits

Istio evaluates authorization actions in a defined order — CUSTOM first, then DENY, then ALLOW.

Which means the external authorizer is consulted before your DENY rules and before your ALLOW rules. A CUSTOM policy that denies ends the request there. A CUSTOM policy that allows does not end anything — the request still has to survive the DENY policies and then find a matching ALLOW, exactly as it would have without the authorizer.

The practical consequence is the one people get wrong: CUSTOM allowing a request does not make it allowed. If a DENY policy matches it, it dies anyway. If there is an ALLOW policy selecting that workload and nothing in it matches, it dies anyway — deny-by-default, the same flip that has caught people in every chapter that touches this API. Your external authorizer is one voice in a committee, and it speaks first, not last.

I did not exercise this order. It is documented, I stated the same thing in the foundations book from the same documentation, and I have not watched a CUSTOM allow get overridden by a DENY with my own eyes. If your design depends on the interaction — and a design that layers a platform-wide DENY under a team-owned CUSTOM absolutely does — build the truth table and run it before you rely on it.

Latency, honestly

You have put a synchronous network call in front of every request that matches your rules. Say that out loud once more, slowly, because the architecture diagram will not.

Every matched request now costs: the proxy’s decision to call out, a connection to the authorizer, the authorizer’s own work — which might include a database query, and if it does, you have put your database on the authorization path of every request — the response, and the proxy resuming. That happens before your service sees the request, on every call, at your busiest moment, and your p99 is the p99 of the slowest link in that chain rather than of your own code.

This is why rules scoping matters and why cache TTLs matter, and it is why the mature implementations push the policy evaluation to a local process with the data pre-loaded rather than making a remote call to a service that makes a remote call to a database.

I have no measurements. I would not trust anybody else’s either — this number depends entirely on what your authorizer does — so measure it yourself, on your topology, before it becomes load-bearing, and measure it under load rather than with a single curl on a quiet cluster.

OPA, and the honest recommendation

The common implementation of the authorizer is Open Policy Agent. Policies are written in Rego, it speaks Envoy’s ext_authz gRPC protocol out of the box, it can run as a sidecar or as a service, and it is the thing most people mean when they say “we do external authorization”. If you are going down this road, that is where you should start looking — not because it is the only option, but because it is the one with the ecosystem, the documentation, and the other people who have already made your mistakes.

And now the recommendation I actually believe, which is less exciting than the chapter it sits at the end of.

Most services do not need this. ext_authz is a chokepoint, a hard dependency, a latency tax, and a second system to run — one that is, by construction, in the blast radius of every request you serve. The cost is real and the failure modes are nasty. Do not reach for it because it is architecturally elegant. Reach for it when you have a specific problem it solves: uniform authorization across services written in languages that do not share a library; a compliance requirement to enforce and audit decisions outside the application; a policy that changes far faster than you can ship code.

If what you have is one service, in one language, with a team that owns it — put the check in the handler, next to the data, where it can be read and tested and where an outage in it takes down exactly one thing. The mesh’s job is to make sure the request that reached that handler is authenticated, encrypted, from a known workload, and carrying a verified user. That is a great deal of work, and it is work the mesh does superbly, and it stops precisely where your domain begins.

What I would test first

Since this chapter is a map, let me at least mark it properly. If I were standing this up on Monday, here is the order I would run things in — and every item is on the list because it is a place where the last eight chapters suggest reality diverges from the documentation.

1. Does the CUSTOM policy actually reach the provider at all? Point provider.name at a provider that does not exist and see what happens. If the answer is “the policy applies cleanly and enforces nothing”, you have found this book’s favourite bug for the fourth time, and you now know to alert on it.
2. Is a waypoint required? ext_authz is L7 — it needs a proxy that can parse the request. In ambient that means the waypoint or the gateway, and everything chapter eight said about a JWT policy landing on a Service with no waypoint applies here without modification. Test the no-waypoint case deliberately, before someone finds it for you.
3. Kill the authorizer. Not gracefully — delete the deployment, mid-traffic, and watch what the requests do. That is the only way you will ever really know which way your mesh fails, and it is a five-minute experiment that answers the most important question in the chapter.
4. The evaluation order. A CUSTOM that allows, under a DENY that matches. Build the truth table and confirm it, rather than trusting me, who is trusting a document.
5. Latency, under load. Not a curl. Load.

None of that is exotic, none of it takes a day, and all of it is the difference between a control you have and a control you have described.

Final thoughts

Stand back and look at what four chapters have built, because the shape is the point.

There are now four places a request can be refused, and they are stacked:

1. ztunnel, at L4, refusing a workload identity it does not recognise.
2. The waypoint, at L7, refusing a method or a path that your policy does not permit.
3. The JWT filter, at L7, refusing a request whose end-user token is invalid — or, with the AuthorizationPolicy from the last chapter, absent.
4. Your own authorizer, refusing a request for a reason only your business understands.

Four layers, four enforcement points, four different processes — and, as the foundations book’s authorization chapter established with a curl in each hand, four different-looking denials. The L4 refusal is a connection reset: http=000, curl exit=56, no HTTP response at all, invisible to any dashboard built on status codes. The L7 refusal is a 403. The JWT refusal is a 401. And the external authorizer’s refusal is whatever your service decided to return, which means the last of the four is the only one whose shape you control — and the only one that can be made to say something useful to the person who hit it.

Learn to read those four denials on sight. When a request fails in a mesh, the form of the failure tells you which of the four layers refused it, and that single observation collapses the search space faster than any amount of log-grepping. A reset is ztunnel. A 403 is a waypoint. A 401 is a token. A 403 with your own error body in it is your authorizer, and now you can go and read its logs.

And keep the boundary honest. Every one of these layers, including the one you wrote yourself, is a proxy making a decision about a request. None of them is your database. None of them knows what an order is, or who owns it, or what your business promised a customer last March. The mesh will carry your authorization decision, enforce it uniformly, and audit it. It will never make it for you. That was true before ambient, it is true now, and it will still be true when this whole architecture has been replaced by something else with a better name.

Next: Two Clusters, One Root CA, and Nowhere to Put the Gateway