A Canary That Counts Requests, Not Pods

Two implementations, the same 90/10 split, and neither one delivered 90/10 — plus the reason a mesh beats a rolling update at canarying, which is not the syntax.

Here are two measurements. Same application, same cluster, same weights — web at 90, web-canary at 10 — and a hundred requests fired at the front door.

In the Kubernetes series, through NGINX Gateway Fabric:

     85 <h1>The Bookshop (SUMMER SALE)
     15 <h1>CANARY BUILD

And now, through the Istio gateway we stood up in the last chapter:

    93 <h1>The Bookshop (SUMMER SALE)
     7 <h1>CANARY BUILD

Fifteen, then seven. On the identical 90/10 configuration. Two conformant, production-grade implementations of the same specification, neither of which delivered anything like ninety and ten — and both of which were behaving exactly as designed.

That is the first half of this chapter, and it is a lesson about numbers. The second half is about why anybody bothers: because a mesh canaries by request, and a Deployment canaries by pod, and those are not two syntaxes for the same idea. They are different capabilities, and once you have seen the difference you will not want to go back.

The route, which you have already written

The HTTPRoute attached to the bookshop’s Istio gateway is the same object, in the same API, that you wrote in the Kubernetes series, with the same parentRefs and the same path matches. Nothing here is Istio-flavoured:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: bookshop-routes
  namespace: bookshop
spec:
  parentRefs:
    - name: bookshop-gw
  rules:
    - matches:
        - path: { type: PathPrefix, value: /books }
      backendRefs:
        - name: catalog
          port: 80
    - matches:
        - path: { type: PathPrefix, value: / }
      backendRefs:
        - name: web
          port: 80

And it does what it did before:

GET /       -> 200
GET /books  -> 200

/ is the storefront’s HTML. /books is the catalog’s JSON, served by the catalog Deployment directly rather than proxied through the storefront. One port, two services, matched by path — by an Envoy this time rather than an nginx, and if I had not told you, the responses would not have.

Everything you learned about HTTPRoute still holds and I am not going to re-teach it: matching is by specificity rather than by document order, so the / rule does not swallow /books even though it would match every request; PathPrefix matches path segments, so /books does not match /booksellers; a rule may carry filters for header rewrites and redirects. Same API, same rules, different implementation — which is the entire promise of a specification, delivered without ceremony.

The one thing that is different is invisible from here, and it is worth a sentence. When this Envoy forwards /books to catalog, that hop is mTLS, because the gateway is a mesh workload with a SPIFFE identity — and if catalog has a waypoint bound, the request passes through it and is subject to whatever L7 policy lives there. Under NGINX Gateway Fabric the same route produced the same 200 over a plaintext pod-network hop from a proxy the mesh had never heard of. Identical YAML, identical status code, different guarantees.

What is new in the YAML starts with a number.

The canary

web runs bookshop:v1 and greets you with The Bookshop (SUMMER SALE). web-canary is a second Deployment running bookshop:v2, with its own Service and a GREETING environment variable set to CANARY BUILD, so that the response body tells you unambiguously which one answered. No shared Service, no label trickery — two backends, two names.

Give the root rule a second backendRef, with weights:

    - matches:
        - path: { type: PathPrefix, value: / }
      backendRefs:
        - name: web
          port: 80
          weight: 90
        - name: web-canary
          port: 80
          weight: 10

Apply it. Nothing restarts. No pod is created, deleted, or rescheduled — the Gateway’s Envoy receives new configuration from istiod and starts making a different choice per request, within a second or two of the kubectl apply returning.

Then fire a hundred requests at the gateway and count the greetings:

    7 <h1>CANARY BUILD
   93 <h1>The Bookshop (SUMMER SALE)

Ninety-three and seven.

Weights are probabilistic, not a quota

Print that number rather than hiding it — the honest version of this experiment is the one that teaches you something, and the flattering version teaches you a falsehood.

The gateway is not keeping a counter and dealing out every tenth request to the canary. It is making an independent, weighted, random choice for each request — a ten-percent coin, flipped a hundred times, with no memory between flips. A hundred flips of a ten-percent coin lands on seven about as readily as it lands on fifteen. The standard deviation of that binomial is three, so anything from four to sixteen is a completely ordinary Tuesday.

You have seen exactly this shape before, and from the layer below. When the Kubernetes series counted which pod answered through a two-replica Service, twenty requests came back twelve and eight — not ten and ten — because kube-proxy in iptables mode picks a backend at random, not round-robin. The sentence that chapter ended on applies verbatim here: over a million requests it converges, and over a hundred it does whatever it likes.

So take the two measurements at the top of this chapter and put them side by side. NGINX Gateway Fabric: 85/15. Istio: 93/7. Neither implementation is more correct than the other; both drew from the same distribution and got different samples, and if I ran each of them again tomorrow I would get two more numbers. The right way to say this in a design review is “approximately 10%”, and the right way to use it is with enough traffic that the approximation stops mattering.

Three practical consequences, and the third is the one that gets people hurt.

Do not tune it. If you observe 7 out of 100 and adjust the weight to 13 to “compensate”, you have fitted a model to noise — and tomorrow’s sample will punish you for it. The number is the number.

Do not read a canary from a small sample. This is the serious one. You ship the canary, send it 10%, watch for ten minutes, see forty requests reach it, note that two of them were 500s, and declare a 5% error rate. You have measured almost nothing — the confidence interval on 2/40 is enormous, and you are about to roll back a good build or promote a bad one on the strength of a coin flip. Weighted routing is a traffic-shaping primitive, not a sampling guarantee. If your canary decision needs statistical power, it needs volume, and no amount of correct YAML will manufacture volume you do not have.

Weights are not percentages. They do not have to sum to 100, and they are not required to be integers out of a hundred. weight: 3 against weight: 1 is a perfectly good 75/25 — the implementation divides by the sum. And weight: 0 does not mean “remove this backend”; it means “this backend is valid, keep it configured, send it nothing” — which is the difference between a paused canary and a deleted one, and it is a field you set rather than a block of YAML you comment out at two in the morning and forget to put back.

There is a fourth consequence that is really a warning about instrumentation. If you are going to judge a canary at all, judge it on its own metrics, split by version — error rate for web-canary against error rate for web, over the same window, with the request counts printed next to them so that a human can see when the sample is too small to mean anything. The mesh gives you exactly this for free, split by source workload and response code, and we spend chapter 12 reading that table. A canary you cannot measure separately is not a canary. It is 10% of your users having a worse day and nobody finding out.

Why this beats a rolling update, and it is not the syntax

Now the part that justifies the whole apparatus, and it is a genuinely different capability rather than a nicer way to write the same thing.

Go back to the Deployments chapter and ask how you would have done this before there was a mesh. You want 10% of traffic on v2. Your tools are a Deployment, a ReplicaSet, and a Service. So: you run some pods on v1 and some pods on v2, put them behind one Service, and let the Service load-balance across all of them.

Look hard at what you have just agreed to — because it is a great deal more than it looks.

Your traffic split is your replica count. To send 10% of requests to v2, one pod in ten must be running v2. If you run three replicas, your available splits are 0%, 33%, 67%, and 100% — there is no 10%, and no field anywhere in Kubernetes that will give you one. To get an actual 10% you must run ten pods, nine of which are only there to make the arithmetic work. Want 1%? Run a hundred pods. The granularity of your traffic control is bounded, from below, by the size of your fleet, and the only way to buy finer control is to buy more machines.

And it is not even a clean split. The Service picks a backend at random per connection, which means “one pod in ten is v2” gives you approximately 10% with exactly the same binomial noise we have been discussing — you have paid for ten pods and you still do not get a quota.

And changing the split is a rollout. Moving from 10% to 50% means scaling ReplicaSets, which means creating and destroying pods, which means the departure race we spent a whole chapter on — the one where kube-proxy has not yet been told the pod is gone and a customer’s request lands on a closed socket. Every adjustment to your traffic split is a fresh opportunity to drop live requests, and every rollback is another one.

And your canary is cold. Roll back by scaling v2 to zero and the pods are destroyed. Roll forward again and you are waiting on image pulls, JVM warmup, connection pools, and readiness probes. The version you are toggling between is not sitting there ready; it is being rebuilt each time you change your mind.

Now the mesh version. One web-canary pod — a single replica, however large your web fleet is — and a number in a YAML file.

  • 10% of traffic on one pod out of forty-one. The traffic share and the replica count are now independent variables. You size the canary for the load you intend to send it, not for the fraction you want to express.
  • 1% is available. So is 0.5%, if you write weight: 995 and weight: 5. There is no fleet size at which the arithmetic runs out.
  • Changing the split restarts nothing. kubectl apply on one HTTPRoute; istiod pushes new config to the Envoy; the next request is decided under the new weights. No pod is created or destroyed, so the dropped-request race never gets a chance to fire.
  • Rollback is instant and warm. Both versions are running the entire time. Setting the canary’s weight to 0 stops the traffic and leaves the pods hot, connection pools full, caches warm — and setting it back to 10 costs one API call and no cold start.

That is the difference, and it is worth stating in one line: a rolling update moves pods and lets the traffic follow; a mesh moves traffic and lets the pods stay still. Everything painful about canarying with Deployments comes from the fact that “how much traffic does this version get” was never a variable you could set — it was an emergent property of how many pods you happened to be running. The HTTPRoute makes it a variable. That is not sugar. That is a different machine.

Promotion, then, is arithmetic on two integers: 90/10 becomes 70/30 becomes 50/50 becomes 0/100 — each step an edit to one object, each applied in under a second, each reversible in under a second. And when you are convinced, you roll bookshop:v2 into the main web Deployment through an ordinary rolling update — with a preStop hook, because that chapter’s lesson has not stopped being true — and delete the canary backendRef. The mesh does not abolish the rollout. It abolishes the need to do the risky part of it in the dark.

Be honest about the price, though, because there is one. You are now running two Deployments, two Services, and two images of the same application, and something has to remember to clean up the second one. Canary infrastructure that is never torn down becomes an environment — a permanently half-promoted v2 that three services now depend on, with its own bugs and no owner. The Deployment-based approach at least had the virtue that a rollout ends. A weighted split will happily sit at 90/10 for a year, and I have seen it do exactly that.

Matching: the other way to pick a version

Weights are the blunt instrument. The sharp one is a match, and it is the basis of a workflow I would recommend over percentage canaries for anything that touches money.

An HTTPRoute rule can match on a header as well as a path:

    - matches:
        - path: { type: PathPrefix, value: / }
          headers:
            - name: x-bookshop-track
              value: canary
      backendRefs:
        - name: web-canary
          port: 80
    - matches:
        - path: { type: PathPrefix, value: / }
      backendRefs:
        - name: web
          port: 80

Anybody sending x-bookshop-track: canary reaches the new build. Everybody else — which is to say every customer, every crawler, every synthetic check — reaches the old one. Your team, your QA, your integration suite, and your own browser with an extension that sets the header can all live on v2 in production, against the real database, with the real traffic around them, at zero risk to anyone who did not opt in. Then, when you are comfortable, you add the weights and start letting strangers in.

I did not run this one. The ground truth in this book covers path matching and weights, both through the Istio gateway, and I have shown you the counts. The header match above is the documented shape of a standard-channel HTTPRoute field, and I have no output from this cluster to put under it — so treat it as a design you should verify rather than a result I measured. I am flagging it because the alternative is to quietly imply I ran everything, and the whole premise of this book is that you can tell the difference.

One thing weights genuinely cannot give you, and it is worth knowing before you promise it to anyone: stickiness. A weighted split decides per request, so a single user browsing the shop can be served by v1, then v2, then v1 again, three clicks in a row. If your two versions differ in a way the user can perceive — a changed layout, a changed session format, a changed cookie — that is not a canary, it is a haunting. Header or cookie matching is how you make a user’s version stable, and a percentage split is how you make the population’s exposure small. They solve different problems and you will eventually want both.

The ambient idiom: an HTTPRoute whose parent is a Service

Everything so far has been north–south: traffic arriving at the gateway from outside. But the bookshop’s interesting traffic is east–west — web calling catalog, orders calling catalog — and none of it goes anywhere near the gateway.

So how do you apply an HTTP rule to a call that one of your pods makes to another?

You write an HTTPRoute whose parentRefs is not a Gateway. It is a Service:

  parentRefs:
    - group: ""
      kind: Service
      name: catalog
      port: 80

And the route is accepted, with the Service as its parent:

route status:
  parent=Service/catalog  Accepted=True

Read that parentRefs carefully, because two of its four fields are easy to get wrong. kind: Service is the whole point — you are attaching this route to a Service rather than to a Gateway. And group: "" is the core API group, which is where Service lives; leave it out or write group: core and you will have a bad afternoon, because the default group for a parentRef is Gateway API’s own. The empty string is not a typo. It is Kubernetes’ name for “the original API group”, and it has been quietly ruining people’s YAML since 2015.

What this does: it says this rule applies to traffic addressed to the catalog Service, no matter who sent it. Not traffic through a gateway — traffic between your own pods, inside the cluster, on the mesh. The traffic that the last eight chapters have been about, and that no ingress route has ever touched.

And here is where the whole architecture snaps together. Who enforces it? The waypoint. That is the entire reason chapter 7 exists. An HTTPRoute attached to a Service is a set of HTTP rules — paths, headers, weights, timeouts — and the only thing in ambient mode that can read an HTTP request is the Envoy you deliberately deployed and bound to that Service. ztunnel cannot help you here; it never sees a request. The route is written against the Service, and it is executed by the Service’s waypoint, and if the Service has no waypoint then there is nothing in the path that can read a path.

That last clause is the one to remember, and it is the same family of failure as the fail-open authorization footgun that this book’s security chapter is built around. I did not run an HTTPRoute against a Service with no waypoint bound, so I cannot tell you exactly what Istio reports when you do — I am not going to invent the status condition. What I can tell you with confidence is where the enforcement lives, because I watched it enforce: bind the waypoint, and the rules take effect; the waypoint is the machine.

This idiom is how you get L7 behaviour on internal traffic, and it is precisely what the next chapter needs. A timeout is an HTTP rule. It goes in an HTTPRoute. That HTTPRoute’s parent is the catalog Service. And the waypoint is what gives up after a second and returns a 504.

What HTTPRoute still cannot do

One honest limit before we go, because it will save you an hour of schema-guessing.

Gateway API’s standard channel — v1.4.0, which is what this book installs — does not have a retry field. Not “it is called something else”. It is not there:

Error: HTTPRoute in version "v1" cannot be handled as a HTTPRoute:
strict decoding error: unknown field "spec.rules[0].retry"

Strict decoding, rejected at the API server. Which means that in ambient mode today, if you want the mesh to retry a failed request on your behalf, you reach for a classic Istio VirtualService — the older, Istio-specific, richer API that Gateway API is on a long road to replacing and has not replaced yet. Timeouts, HTTPRoute has. Retries, it does not — and the schema will tell you so rudely rather than silently, which is the one mercy on offer. Both are next chapter’s problem, and I raise it here so that when you go looking for retry: in the HTTPRoute schema and cannot find it, you know it is not you.

The general shape is worth carrying: Gateway API is the future and it is not finished. Use it where it reaches, and reach for VirtualService and DestinationRule where it does not — without embarrassment, and without pretending the standard covers more than it does.

Final thoughts

The number 93/7 is the most valuable thing in this chapter, and it is valuable precisely because it is ugly. A tutorial that ran a hundred requests through a 90/10 split and got 89/11 would print it and let you believe that weights are a quota. Ours got 93/7, and the Kubernetes series got 85/15, and the truth those two numbers tell together is that you are looking at a random process and you should treat every canary metric you compute from a small sample with the suspicion it deserves. Traffic splitting gives you a dial, not a promise — the dial is honest about the long run and says nothing at all about your next hundred requests.

But the reason to have the dial is the thing I would want you to argue for in your own organization. Before the mesh, “what fraction of traffic sees the new version” was not a question your platform could answer, because the answer was a side effect of your replica count, your Service’s random load balancing, and the timing of a rolling update. You could not set it. You could only arrange for it to be approximately true and then hope, and every adjustment cost you a rollout, and the rollout is the thing that drops requests.

Now it is a number in a file. Ten percent, on one pod, changed in a second, reverted in a second, with both versions warm the entire time. That decoupling — traffic share from replica count, and traffic change from pod churn — is the first thing in this book that a mesh gives you which you genuinely could not build with what Kubernetes ships. Everything before it was better mTLS or better policy or better metrics than you had. This is a capability that was not on the menu.

And it sets up the next one perfectly. If the mesh can decide where a request goes, it can also decide what to do when the place it went does not answer — or answers slowly, or answers with a 500, or keeps passing its readiness probe while failing every single request it receives. That last one is real, we did it on purpose, and Kubernetes never noticed.

Next: Ready, Running, and Serving Nothing but 500s

Comments