The 503 That catalog Never Sent

Fault injection, delay injection, and header-based routing — how to break a dependency on purpose without touching it, and why a weighted split can never send *your* traffic to the canary.

Sixty requests to the catalog, and sixty of them succeeded. Zero failures. The service is healthy, the pods are Running, the readiness probes are green, and the shop is serving books.

Now apply eleven lines of YAML — no rebuild, no rollout, no restart, no change to a single byte of the application — and fire two hundred more:

aborted 95 / 200 = 47%

Ninety-five 503s. The catalog is still healthy — its pods have not restarted, its logs are empty, its own request counter never moved, and if you exec into a catalog pod and curl it directly it will answer every time. Ninety-five requests were refused by something that is not the catalog, on behalf of the catalog, without ever telling the catalog they existed.

That is fault injection, and it is the most useful thing in this chapter, because it is the only honest way to answer a question every service in your system is currently failing to answer: what does the caller do when this dependency breaks?

You have code for that — everybody does. There is a timeout in there somewhere, and a retry loop, and an error branch that logs something and returns a friendly message, and a circuit breaker that somebody added after an incident in 2023. None of it has ever run in production against a real failure. It has been reviewed. It has not been tested — because testing it would mean breaking the dependency, and breaking the dependency means breaking the dependency.

The mesh dissolves that trade. It can manufacture the failure in the request path — in front of a service that remains perfectly healthy, for a percentage of traffic you choose — and take it away again with kubectl delete.

The API, briefly

Everything here is VirtualService, the classic Istio API we took apart in the last chapter. It is worth restating why, because it is the reason this chapter is not written against Gateway API like the routing chapter in the previous book was.

Gateway API is the future, it is where Istio is going, and for the ordinary work — a hostname, a path prefix, a weighted split — it is the better object and you should use it. But its standard channel has no fault injection, and no retries. Not “a different spelling of them” — no field at all. The fault-injection, delay, mirroring, and retry vocabulary that this chapter is built on lives in VirtualService, and it will keep living there until Gateway API’s experimental channel grows equivalents and they graduate. So: Gateway API for the shape of your traffic — VirtualService for the experiments you run on it. Two objects, in the same cluster, doing different jobs. That is not a transitional embarrassment, it is the current state of the art, and pretending otherwise gets you a chapter of theory and no numbers.

One structural note that matters more in ambient than it did in the sidecar world, and it is the same seam as the authorization fail-open: everything in this chapter is L7, so everything in this chapter needs a waypoint. A VirtualService that inspects an HTTP header, or manufactures an HTTP 503, or holds a request for two seconds, is asking something in the path to open the request and act on it. ztunnel cannot do that — it is an L4 proxy that moves bytes and does not parse HTTP, which is why it cannot send you a 403 and why it cannot send you a 503 either. The bookshop has a waypoint bound to the catalog Service, which is why every number below exists. Bind these rules to a Service with no waypoint and you get the same silent nothing that the L7 authorization policy got: a well-formed object, accepted by the API server, enforced by nobody. I did not run that specific case for fault injection, so I am telling you the mechanism rather than showing you the output — but it is the same mechanism, and .status.conditions and istioctl analyze are the same two places you check.

Aborting requests that were never sent

Here is the object. hosts: [catalog] means “requests to the catalog Service”, and the fault block sits in front of the route rather than replacing it:

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: catalog
  namespace: bookshop
spec:
  hosts:
    - catalog
  http:
    - fault:
        abort:
          httpStatus: 503
          percentage:
            value: 50
      route:
        - destination:
            host: catalog

Read the shape carefully, because it explains the behaviour. The route is still there and still points at the real catalog. The fault is a filter in front of that route. For each request the waypoint rolls a fifty-percent die — on a hit it composes a 503 itself, returns it to the caller immediately, and the route is never taken; on a miss the request proceeds to a real catalog pod as if the VirtualService were not there at all.

So the catalog does not fail. The catalog is not asked. The 503 is manufactured by an Envoy that is standing in the request path on the catalog’s behalf — and it is manufactured before the request goes anywhere. This is not “make the service unhealthy”; it is “make the service look unhealthy, to its callers, on the wire, exactly as an unhealthy service would look — and leave the service alone.”

That distinction is why fault injection is safe to run in an environment you care about. Nothing is broken. Nothing has to be repaired afterwards. You delete the object and the next request is fine — there is no recovery, because there was no damage.

Two runs, with a clean baseline first:

baseline (no VirtualService)   0 failures / 60 requests
percentage 50                  aborted  95 / 200 = 47%
percentage 20                  aborted  43 / 200 = 21%

It does what it says. Forty-seven against a requested fifty, twenty-one against a requested twenty — and those gaps are not error, they are the same coin the weighted-canary chapter measured at 93/7 when it asked for 90/10. The percentage is a per-request independent draw — not a quota. Two hundred draws from a fifty-percent coin lands on 95 about as readily as it lands on 105, and if you rerun it tomorrow you will get a third number. Do not tune the percentage to make the measurement come out round.

The baseline is not a formality

Notice that I ran sixty requests with no fault injected and confirmed zero failures before touching the percentage. That step looks like padding — it is not, and I know it is not, because I skipped it the first time.

My first measurements came out at 70% aborted for a requested 50, and 44% for a requested 20 — systematically wrong, wrong in the same direction, and far too wrong to be sampling noise. I spent an unreasonable amount of time reading Envoy config and being suspicious of Istio. Istio was innocent. A catalog pod I had deliberately poisoned during an earlier experiment was still poisoned — and I had just deleted the DestinationRule that had been ejecting it from the load-balancing pool. It was back in rotation, failing about a third of everything, underneath the injected aborts. The arithmetic fits exactly — 1 − (0.5 × 0.67) = 66%, 1 − (0.8 × 0.67) = 46% — and it fits because it was never a mesh problem at all.

That story is the debugging chapter’s to tell properly, and I will not spend it here. The operational rule it produces belongs in this one: a mesh measurement is only as clean as the cluster underneath it, and a stale experiment is the easiest way in the world to publish a wrong number. Run the baseline. Every time. If the baseline is not zero, you are not measuring what you think you are measuring, and the number you take to the design review will be a fiction with a decimal point in it.

Making a fast service slow

The abort is the dramatic one. The delay is the useful one.

    - fault:
        delay:
          fixedDelay: 2s
          percentage:
            value: 100
      route:
        - destination:
            host: catalog

The catalog answers in single-digit milliseconds. With that object applied:

request took 2.008357s

Two seconds and eight milliseconds. The eight milliseconds is the catalog actually doing its job — the two seconds is a waypoint holding the request in a queue and doing nothing at all, on purpose, because you asked it to.

You have just made a dependency slow without touching the dependency. Think about what that unlocks, because it is not obvious until you have needed it.

Every timeout in your codebase is a guess. Somebody picked a number — 30 seconds, because that was the library default; 5 seconds, because it felt about right; 1 second, because a staff engineer said so in a review — and then nobody ever produced a request that took longer than the number, so nobody ever found out whether the timeout fires cleanly, whether the error branch below it works, whether the retry that follows it makes the situation better or considerably worse, or whether the whole call chain simply blocks a thread and takes the front door down with it. Timeout code is the least-exercised code in most systems and the most load-bearing when things go wrong — and that is not a coincidence. It only runs on the worst day, which is the worst possible day to discover a bug in it.

fault.delay is how you produce the worst day on a Tuesday afternoon. Set the delay to just over your timeout and watch what your caller does. Then set it to just under your timeout and watch again — because a request that takes 900ms against a 1s timeout is the far nastier case. It succeeds, slowly, and your connection pool fills up with successes.

This composes with everything the previous book set up. The waypoint timeout from the resilience chapter turned a slow request into a 504 at exactly one second; a fault.delay of two seconds is precisely the input that timeout was written for, and putting the two objects in the same cluster is a complete, self-contained, mesh-level test of a mesh-level control. I did not run that combination — I measured the delay against an untimed path and I measured the timeout against a slow endpoint, separately — so I am describing the experiment rather than reporting it. It is an obvious one to run, it takes four minutes, and it is exactly the sort of thing you should be doing before you trust a timeout you inherited.

A few things worth knowing about delays, none of which I measured:

  • fixedDelay is documented to accept a duration string — 2s, 500ms — and there is also an exponentialDelay field in the API that is marked as not implemented in the reference. I used fixedDelay. Use fixedDelay.
  • A delay costs the proxy a held connection — and a hundred percent delay across a busy service means a lot of them, all at once, in a pod with finite memory. On a bookshop with a curl loop this is invisible. On a production service at real concurrency it is a load test of your waypoint that you did not intend to run — start with a low percentage.
  • Combine delay and abort in one fault block and Envoy applies the delay first, then the abort — a slow failure, which is the shape most real outages actually have. Documented; not run.

The header, and the thing a percentage can never do

Now the part of this chapter that changes how you ship software.

no header       ->  <h1>The Bookshop (SUMMER SALE)
x-canary: yes   ->  <h1>CANARY BUILD

Two requests, same URL, same cluster, same second. One of them carries a header and gets a different build of the application. And this is the object that did it:

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: web
  namespace: bookshop
spec:
  hosts:
    - web
  http:
    - match:
        - headers:
            x-canary:
              exact: "yes"
      route:
        - destination:
            host: web-canary
    - route:
        - destination:
            host: web

The http list is ordered, and the first matching rule wins. The header rule is first and it is specific. The last rule has no match at all, which makes it the default — the catch-all every request falls through to when nothing above it matched. Put the catch-all first and it will swallow everything, canary traffic included, and the header rule below it will never be evaluated. This is not a subtlety — it is a routing table, and routing tables have always worked this way.

Now the payoff, and it is the sentence to take out of this chapter:

A weighted split fundamentally cannot do this.

Go back to the canary chapter of the previous book. We sent 10% of traffic to web-canary by weight and measured 93/7 — and the weights worked exactly as designed, as an independent, weighted, random choice per request. But that mechanism — a coin flipped once per request, with no memory and no notion of who is asking — has no way to express “the ten percent that is me.” There is no field for it. There is no weight you can set that means “my browser”. The coin does not know your name, and asking a probability distribution to pick a specific person is not a configuration problem — it is a category error.

So a weighted canary can only ever answer one question: if I expose a small random slice of my users to this build, does the aggregate error rate go up? That is a real question and it is worth asking. It is not the question you actually want to ask first — which is: does this build work at all, against production data, in the production cluster, with production dependencies, before a single stranger touches it?

Header routing answers that one. And the answer costs you nothing, because one hundred percent of your real users are still on stable. Not 90% — all of them. The canary receives exactly the traffic that opts into it and no other traffic exists for it, so the blast radius of a catastrophically broken build is: you.

That is what a dark launch is, and that is what internal dogfooding is. Your team sets the header in a browser extension and lives on v2 in production for a week. Your integration suite sets the header and runs the whole regression pack against v2 — in the real cluster, against the real catalog, while the shop serves customers from v1 two hops away. Your on-call sets the header to reproduce a bug on the new build without exposing it. And when you are satisfied, then you add the weights and start letting strangers in at 1% — having already answered every question that could have been answered without them.

The previous book’s routing chapter showed you a header match written as a Gateway API HTTPRoute and told you honestly that it had not been run. Here it is, run, on the classic API, with the output above. The design was right. Now it is also measured.

exact is one of three documented match forms — the others are prefix and regex, and the same match block can also condition on uri, method, scheme, port, queryParams, and sourceLabels, which is a genuinely powerful clause and is worth reading the reference for. I ran exact on a header. The rest are documented shapes I am pointing at, not results I measured.

Cookies, and stickiness

A header is perfect for a machine and terrible for a human. Your QA engineer will not set x-canary: yes on every request from a real browser, and neither will the twenty internal users you want on the new build for a fortnight.

The documented answer is that a cookie is just a header, and VirtualService will match on it:

    - match:
        - headers:
            cookie:
              regex: "^(.*;?)?(canary=yes)(;.*)?$"
      route:
        - destination:
            host: web-canary

Set the cookie once — at a login page, a feature-flag endpoint, an internal /enroll route — and that user is now stably on the canary for as long as the cookie lives: every request, every click, every asset, across sessions. Which is the thing a weighted split cannot give you either, in a second and different way. A percentage decides per request, so a single user browsing the shop can be served v1, then v2, then v1 again, three clicks in a row. If the two builds differ in any way a person can perceive — a layout, a session format, a serialised cookie shape — that is not a canary, it is a haunting.

I did not run the cookie match. The regex above is the documented idiom — the cookie header holds all the cookies, semicolon-separated, so you match a substring of it rather than the whole value — and it is the shape you will find in Istio’s own examples, and I have no output from this cluster to put underneath it. Verify it before you rely on it. Note also that a regex on a header is a regex evaluated on the hot path of every request through the waypoint: fine at bookshop scale, worth thinking about at yours.

There is a second, related mechanism I also did not run and should name so you know it exists — DestinationRule consistent-hash load balancing (trafficPolicy.loadBalancer.consistentHash, keyed on a cookie, a header, or the source IP), which pins a given user to a given pod rather than to a given version. That solves session affinity, which is a different problem from cohort routing and is constantly confused with it. Cohort routing decides which build. Consistent hashing decides which replica of it. You may want both — they are not substitutes.

Mirroring: run the new build on real traffic and throw the answer away

The last capability in the routing vocabulary is the strangest and, on a large system, the most valuable. I did not run it, and everything in this section is documented behaviour I am describing rather than output I measured — flagging that up front rather than at the end, because it would be very easy to read this section as another result.

  http:
    - route:
        - destination:
            host: catalog
      mirror:
        host: catalog-v2
      mirrorPercentage:
        value: 10.0

The route is unchanged — every request goes to the real catalog, and the real catalog’s response is the one the caller gets. The mirror sends a copy of ten percent of those requests to catalog-v2, fire-and-forget. The waypoint does not wait for the mirrored response, does not time it into the caller’s latency, and discards it entirely. The mirrored service is being sent live production traffic and its answers are going straight into the void.

Which sounds pointless until you notice what it buys. The new build is being exercised by real requests, with real shapes, in the real cluster — and you can watch its metrics (error rate, latency histogram, memory, logs) under genuine production load, with zero user-visible risk, because no user is ever shown anything it produced. Synthetic load tests approximate the traffic you imagined. Mirroring gives you the traffic you have — including the malformed request from the client somebody wrote in 2019 that nobody remembers.

Two things about it that are documented and that you must think about before you switch it on:

  • The mirrored request has side effects. It is a real HTTP request to a real service. If catalog-v2 writes to the same database, you have just doubled ten percent of your writes. If it charges a card, it has charged the card. Mirroring is safe on a read path, and safe on a write path pointed at a scratch datastore — and it is a live-fire incident for anything else. The mesh cannot know which you have.
  • Istio appends -shadow to the mirrored request’s Host/Authority header — so catalog becomes catalog-shadow — which exists precisely so that a service can tell it is being shadowed and behave accordingly. Which means the honest way to run a mirror on a write path is to make the application read that header and skip its side effects. Which means mirroring, alone among everything in this chapter, is not free of code changes. Right at the end of a chapter about things you can do without touching the app, there is one that requires touching the app — and it is a small rehearsal for the tracing chapter, where that bill comes due properly.

Final thoughts

Every organisation I have worked in has a document somewhere describing what happens when a critical dependency fails. It has a diagram in it. It has been through a review. And it is almost always wrong, in a specific, boring, recurring way — the code it describes has never once executed.

The reason is not laziness. It is that until quite recently there was no safe way to run it. Testing your retry logic meant breaking the catalog, and breaking the catalog meant an incident, so you built a staging environment where you were allowed to break things — and the staging environment had one replica, no traffic, a different database, a tenth of the latency and none of the concurrency, so it answered a question you had not asked. The failure modes that matter are the ones that only appear under real load against a real system, and the only place a real system exists is production.

Fault injection is the first mechanism I have used that resolves that honestly, and it does it by moving the failure out of the service and into the path. The catalog is never broken. There is nothing to repair, nothing to roll back, nothing to explain to the team that owns it. There is an object in etcd that makes half the requests fail — and when you delete the object the failures stop, immediately and completely, with no recovery to wait for. The blast radius is bounded by a percentage you chose and terminated by a kubectl delete you can type in three seconds. That is not a testing strategy that got braver; it is a testing strategy that got cheaper to abandon — a much more useful property, and the reason it will actually get used.

The header match is the other half of the same idea, arrived at from the opposite direction. Fault injection lets you break production without hurting anyone. The header route lets you ship to production without exposing anyone. Both work because the mesh has put something in the request path that can be told to lie — and both are impossible with a percentage, because a percentage can only ever hand you a random slice of strangers, and a random slice of strangers is exactly who you do not want going first.

The engineer with the header set in their browser, using the new build in production against the real database, at nine on a Tuesday morning, is running the most valuable test you have. Give them a way to be first.

Next: The Log Line Nobody Wrote

Comments