A Bad Token Gets 401. No Token Gets In.

Here is a RequestAuthentication. It names an issuer, it points at that issuer’s public keys, and it applies cleanly:

$ kubectl apply -f catalog-jwt.yaml
requestauthentication.security.istio.io/catalog-jwt created

Now test it the way everybody tests it. Send a rubbish token — a string that is not a valid JWT, or one signed by a key the issuer never held — and see whether the mesh notices:

a GARBAGE token  -> http=401

401. Unauthorized. The mesh cracked open the request, found the Authorization header, tried to verify the signature against the issuer’s JWKS, failed, and refused — no code changed, no library imported, no service redeployed. It works. Ship it, close the ticket, and put “end-user authentication enforced at the mesh” in the design doc.

Now send nothing at all. No Authorization header. No token. The request a completely unauthenticated stranger would send:

no token at all  -> http=200

Two hundred. Straight through.

The RequestAuthentication is still there. It is still valid. It is still, in the only sense that matters to it, working perfectly — because RequestAuthentication validates a token if one is present, and it does not require one. It is a validator, not a gate. And the way people find this out is almost never by reading that sentence in the documentation, where it does in fact appear. They find out the way you would expect: by testing the case that fails, seeing it fail, and never testing the case that succeeds for the wrong reason.

If you take one thing from this chapter, take this: the interesting test of an authentication control is not “does a bad credential get rejected”. It is “does no credential get rejected”. Those are different questions, they have different answers here, and only one of them is the one an attacker will ask.

Two kinds of identity, and they are not the same kind

Before we fix it, here is the distinction that organises everything in this chapter — because the mistake above is downstream of a conceptual muddle rather than of a typo, and if the muddle survives, you will make the mistake again in a different shape.

The foundations book’s authorization chapter was entirely about service identity. A pod has a ServiceAccount, the ServiceAccount becomes a SPIFFE identity, the identity is baked into a 24-hour X.509 leaf, and the leaf is presented on every mTLS connection. The claim being made is “I am the orders service.” The mesh verifies that claim cryptographically, and an AuthorizationPolicy can then say: only that identity may open a connection to the catalog. That is the principals field, ztunnel enforces it, and a denial is a connection reset.

This chapter is about end-user identity. A human logs into your bookshop, your identity provider hands them a JWT, their browser sends it up with every request, and the claim being made is “I am user 42, and I am in the admin group.” Nothing about that claim is related to the first one. The pod carrying the request still has its own SPIFFE identity; the JWT rides inside the request the pod is making. They are two independent assertions, verified by different machinery, against different roots of trust, describing different subjects.

They are so different that Istio gives them different fields in the same policy — principals for the workload, requestPrincipals for the end user — and you can, and often should, use both in one rule. Conflating them is how people ship holes. “The mesh authenticates everything” is true and useless; the question is what it is authenticating, and a mesh that has perfectly established that the request arrived from the web ServiceAccount has established nothing at all about who is sitting at the other end of the browser session.

What RequestAuthentication actually says

The object is small, and every field in it is load-bearing.

apiVersion: security.istio.io/v1
kind: RequestAuthentication
metadata:
  name: catalog-jwt
  namespace: bookshop
spec:
  targetRefs:
    - group: ""
      kind: Service
      name: catalog
  jwtRules:
    - issuer: "[email protected]"
      jwksUri: "https://raw.githubusercontent.com/istio/istio/release-1.30/security/tools/jwt/samples/jwks.json"

I used Istio’s own test issuer for this, which is the sane way to learn it — Istio ships a JWT sample under samples/jwt-server, with a well-known test issuer ([email protected]) and a public JWKS you can point at without standing up an identity provider first. In production you swap in your own issuer and its JWKS endpoint — Auth0, Okta, Keycloak, Cognito, your own service, it does not matter — and nothing else about the shape changes.

issuer is the string that must appear in the token’s iss claim. If the token says it was issued by somebody else, no rule matches it and the token is not validated by this rule — which, as we are about to see in painful detail, is not the same thing as the token being rejected.

jwksUri is where Istio fetches the issuer’s public keys. This is the piece that makes JWT verification cheap enough to do on every request: the signature check is local, asymmetric-key arithmetic against a key set the proxy has already fetched and cached. There is no callback to your identity provider on the request path. (You can also inline the key set with a jwks field instead of a URI. I did not run that variant.)

audiences, which I have left out above, is the list of aud values the token must carry. Leaving it out means any audience is accepted, and that is a wider door than most people intend — a token minted for a completely different service of yours, by the same issuer, will validate here. If your identity provider issues audience-scoped tokens, scope them, and say so in the policy. I did not run an audiences mismatch, so treat the exact behaviour as documented rather than demonstrated; the field is there and it means what it says.

That is a complete, correct, useful object. It will verify signatures, check expiry, check the issuer, and reject anything malformed with a 401. And it is, on its own, not a security control.

The footgun, stated precisely

Here are the three measurements, from a real cluster, in order:

RequestAuthentication only, no token at all  -> http=200
RequestAuthentication only, a GARBAGE token  -> http=401

And after adding the second object we are about to write:

AuthorizationPolicy in place, no token       -> http=403

Those three lines are what I measured, and they are all I measured — I did not record the fourth cell, the one where a valid token from the test issuer is presented and allowed through. That is the happy path, it is the only cell everybody assumes, and it is the least interesting of the four. The three above are the ones that teach you something.

Look at the first two together, because the pair is the trap and neither line is remarkable alone.

A garbage token gets 401 because a token was present, a rule matched its issuer, the signature check ran, and the check failed. The machinery did its job.

No token gets 200 because no token means there was nothing to validate, and a validator with nothing to validate has no opinion. The request carries no Authorization header, no jwtRule is triggered, no verification is attempted, and the request proceeds exactly as it would have if you had never written the policy at all. The mesh is not being permissive as a decision. It is being silent, because you asked it a question about tokens and there was no token.

That behaviour is deliberate, and — this is the part that makes it defensible rather than merely dangerous — it is what lets you run a service with optional authentication. A public catalog endpoint that anyone may read, which shows extra fields to a logged-in user: you want anonymous requests to work, and you want the ones that do carry a token to have that token verified so your application can trust the claims. RequestAuthentication alone gives you exactly that, and it is a coherent thing to want.

The problem is that it is also the default thing you get, it looks identical to a working gate under the only test most people run, and the failure is — again, and I am starting to feel like this book has a theme — open.

Imagine the review. Someone writes the RequestAuthentication. Someone else reads it: it names the right issuer, it points at the right JWKS, it is bound to the right Service. Approved. Someone tests it by pasting a mangled token into curl and gets a satisfying 401. Ticket closed — and the catalog is readable by anybody who can reach it, forever, and every single artefact of that process says otherwise.

The other half: AuthorizationPolicy requires

RequestAuthentication validates. AuthorizationPolicy requires. You need both objects, always, unless anonymous access is a decision you have consciously made.

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: catalog-require-jwt
  namespace: bookshop
spec:
  targetRefs:
    - group: ""
      kind: Service
      name: catalog
  action: ALLOW
  rules:
    - from:
        - source:
            requestPrincipals: ["[email protected]/[email protected]"]

And with that applied, the request that sailed through a moment ago:

no token, WITH the AuthorizationPolicy  -> http=403

403. Now it is a gate.

The mechanism is worth being exact about, because it is the same deny-by-default flip the foundations book made a fuss over, arriving from a new direction. The moment an ALLOW policy selects a workload, that workload becomes deny-by-default: a request is permitted if and only if it matches at least one rule in at least one applicable ALLOW policy. This policy has exactly one rule, and that rule requires a requestPrincipal. A request with no token has no request principal — the field is empty, not merely wrong — so it matches no rule, so it is denied. The 403 is not the JWT machinery refusing anything. The JWT machinery still has nothing to look at. It is the authorization layer refusing a request that failed to prove anything about who sent it.

Notice, too, that the two objects produce two different status codes, and the difference is informative rather than cosmetic:

• 401 — a token was present and did not verify. That is authentication failing.
• 403 — the request was not authorized, and in this policy the reason is that it carried no verified end-user identity at all.

If you are debugging this at three in the morning, that distinction tells you which of the two objects is talking to you. A 401 means the token reached the JWT filter and lost. A 403 means the request never had a principal for the policy to accept.

requestPrincipals and principals, side by side

The requestPrincipals value looks strange the first time, and its format is the whole idea in a string:

[email protected]/[email protected]

That is <issuer>/<subject> — the iss claim, a slash, the sub claim. It reads oddly here only because Istio’s test issuer uses the same string for both. In your system it will look like https://accounts.example.com/user-42, and the shape tells you what it means: a subject, as asserted by a particular issuer. A subject without an issuer is meaningless, because anybody can claim to be user 42; the issuer is the part that makes the claim worth anything.

Now put the two identity fields in one policy and read them together, because this is the sentence I actually want you to leave with:

  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/bookshop/sa/web"]
            requestPrincipals: ["https://accounts.example.com/*"]

Two from conditions in the same source block. Both must hold. And they are asking utterly different questions:

• principals — which workload opened this connection? Proved by the mTLS client certificate, a SPIFFE URI, rooted in the mesh’s own CA, and true regardless of what any HTTP header says. web cannot lie about this without stealing a private key.
• requestPrincipals — which end user does this request claim to be from? Proved by a JWT signature, rooted in your identity provider’s keys, carried in a header that the calling workload put there.

Together they say: the web service, acting on behalf of some authenticated end user of ours. That is a genuinely strong statement, and it is one you could not make with either half alone. The workload identity stops a compromised debug pod from calling the catalog at all, no matter what token it forged from a real login. The end-user identity stops web from calling the catalog on behalf of nobody — which is exactly what a server-side request forgery looks like from the inside.

principals is enforced by ztunnel at L4. requestPrincipals requires reading an HTTP header, which brings us to the third footgun, and it is one you have already met.

Where is this actually enforced?

A JWT is an HTTP header. ztunnel cannot read HTTP headers. Follow that through and you land somewhere uncomfortable.

Everything in this chapter is L7. Verifying a JWT means parsing a request, finding a header, splitting it on dots, base64-decoding it, checking a signature, and reading claims. That is Envoy work. In ambient, the Envoys are the waypoint (for east-west traffic to a mesh Service) and the gateway (for north-south traffic arriving from outside). Those are the two places an end-user auth policy can possibly be enforced, and if neither is in the request path, then — exactly as with the L7 authorization rule that opened chapter eleven of the foundations book — you have written a control that nothing is running.

The tell would be the same one that chapter taught you to look for: .status.conditions on the object, WaypointAccepted: False, reason: AncestorNotBound, and an IST0171 from istioctl analyze, while kubectl get prints a perfectly healthy row.

Let me be careful about what I am claiming here, because this is a chapter about people believing things they did not test. I ran the JWT results above with the bookshop’s waypoint in place, which is why they are L7 numbers — 401s and 403s, not connection resets. I did not run the no-waypoint control for RequestAuthentication specifically. What I am telling you is a mechanism (ztunnel is L4, a JWT lives at L7) combined with a measured result from the previous book (an L7-bound policy with no waypoint enforces nothing and fails open). I am confident in the conclusion and I did not measure this exact cell — so before you rely on end-user auth in an ambient mesh, curl the Service with no token from a pod and confirm you get a 403. Which is, you will notice, the same ten-second habit the egress chapter just spent four thousand words arguing for. This is not a coincidence. It is the shape of the whole system.

For north-south traffic — a browser hitting your gateway — the gateway is an Envoy, it is unavoidably in the path, and it is the natural place to terminate end-user auth. Reject the tokenless request at the front door, and everything behind it is dealing with requests that at least carry a verified identity. That does not excuse you from policy on the inside, because the whole point of a mesh is that the inside is not trusted either — but it is the right first cut, and it is where I would start.

There is a migration hazard hiding in this, and after the last chapter you should be able to see it coming. In a sidecar mesh, every pod has an Envoy. A RequestAuthentication bound to a workload is therefore enforced automatically, everywhere, because the enforcer was installed whether you wanted it or not — that is the deal you struck when you paid for a proxy per pod. In ambient, the L7 enforcer is opt-in, per-Service, and you deploy it deliberately. So a JWT policy that was doing real work under sidecars can land in an ambient mesh, on a Service with no waypoint, and enforce nothing at all. Same object. Same YAML. Same review. Different data plane, different enforcer, different outcome — and no error anywhere. I did not run that specific migration, so I am reasoning from mechanism again. But it is the identical shape to the egress regression, and I would be astonished if it did not bite somebody. Check your waypoints before you trust your JWT policies.

Claims as conditions

Once the token is verified, its claims are available to policy, which is where end-user auth stops being a gate and starts being interesting:

  rules:
    - from:
        - source:
            requestPrincipals: ["https://accounts.example.com/*"]
      to:
        - operation:
            methods: ["DELETE"]
      when:
        - key: request.auth.claims[groups]
          values: ["admin"]

Only an authenticated user whose token carries groups: admin may issue a DELETE. Everyone else — authenticated, valid, perfectly legitimate users of your bookshop — gets a 403 on that verb, decided in a proxy, with no code in the catalog service having any idea the rule exists.

I did not run this. The when clause with request.auth.claims[...] is not in my ground truth; my lab covered the presence-and-validity results above and stopped there. The syntax is documented, the mechanism is a straightforward consequence of the verified parts (the waypoint has already parsed and validated the token, so the claims are sitting right there), and I would expect it to work — but “I would expect it to work” is the exact phrase that produced every other finding in this book, so I am flagging it and you should test it. Note also that a claim’s type matters — string versus list versus nested object — and claim shapes vary by identity provider, so the first thing to check is not whether the syntax works but whether it matches the token your provider actually mints.

Related, and likewise not run by me: forwardOriginalToken, which controls whether the JWT is passed on to the upstream service after validation. Your application usually still wants it — the user’s ID has to come from somewhere — so this one matters more than it sounds.

The line the mesh cannot cross

Now the honest boundary, and it is the reason there is a chapter nine.

The mesh can do exactly three things with an end-user identity, and it can do all three well:

1. Verify the token’s signature against the issuer’s keys, so a forged token is rejected before your code sees it.
2. Require that a token is present, so an anonymous request never reaches your service at all.
3. Match coarse claims — a group, a scope, a role — against the method and path of the request.

And here is what it cannot do, ever, in any configuration, at any level of cleverness:

It cannot decide whether this user may edit that order.

Look hard at that, because it is not a limitation that better software will fix. The mesh knows the request is PATCH /orders/8821, and it knows the token says sub: user-42. To make the decision, something must know that order 8821 belongs to user 41, not user 42 — and that fact is not in the request, not in the token, and not in any header. It is a row in your database. It is your domain. The proxy would have to query your data model to evaluate it, and at that point the proxy is your application.

This is the boundary, and I want you to be able to draw it precisely, because teams waste months on the wrong side of it:

The mesh authenticates. The mesh authorizes the shape of a request. Your application authorizes the content.

Coarse-grained, request-shaped, identity-and-verb decisions belong in the mesh — they are uniform, they are policy, they benefit enormously from being enforced outside the code, and they are the same in every language your fleet is written in. Fine-grained, object-level, “does this user own this row” decisions belong in your service, next to the data that makes them answerable, and they always will.

The gap between the two is real, and it is uncomfortable, and it is where most authorization bugs in real systems actually live. Chapter nine is about the escape hatch Istio offers for it: ext_authz, which lets the mesh call out to a service you wrote and enforce whatever that service decides. It is not a way of moving the boundary. It is a way of putting your own code on the mesh’s side of it.

Final thoughts

The RequestAuthentication footgun is not, on its face, in the same league as the egress regression from the last chapter. It is documented behaviour, it has a legitimate use, and the fix is one extra object that any competent review should catch.

And yet I have watched it ship more times than the exotic failures, and I think the reason is worth naming, because it is about how we test rather than about JWTs.

We test the failure we imagined. We write an authentication control while picturing an attacker holding a bad credential — a stolen token, an expired one, one signed with the wrong key — because that is what “authentication” makes us picture. So we craft a bad token, we send it, we get a 401, and the mental model is confirmed. What we did not picture, because it is too stupid to picture, is an attacker who simply does not bother. Sends nothing. Omits the header entirely. And that is the case that walked straight through, because our control was built to answer a question and the attacker declined to ask it.

Almost every authentication hole I have seen has that shape. It is not a cryptographic break. It is a code path where the credential check was skipped rather than failed — an unset header, a null that compared equal, an if (token) with no else. The check is fine. The check was never reached.

So: test the empty case. Test no header, no token, no cookie, no session — not the corrupted credential, the absent one. It takes one curl. It is the difference between an authentication control and a signature-checking service that anybody may simply walk past — and it is, once again, ten seconds.

And then, when it correctly returns 403, and you are feeling good about it, remember that the mesh has still not the faintest idea which orders belong to which customer. It never will.

Next: The Mesh Doesn’t Know What an Order Is