The Front Door Kubernetes Doesn't Ship

Gateway API is CRDs and a controller — and a Gateway with no controller is a beautiful, inert YAML file. Then real traffic, a real canary, and an honest look at Ingress.

The bookshop has been reachable this whole time on http://localhost:30080, and every time you type that port number you should feel a small twinge. Thirty-thousand-eighty. It is a cluster-wide number, allocated out of a range the API server hands out, and it is nailed to a Service. It has no idea what a hostname is, cannot tell /books from /orders, and terminates no TLS. If you want the catalog on the same address as the storefront, a NodePort’s only answer is: pick a second port number.

Then the request arrives. Marketing wants shop.example.com on 443 with a real certificate. The catalog team wants /books to reach their service without going through the storefront. And someone wants to send 10% of traffic to the new build for a day before committing to it. A NodePort answers none of those questions, because a NodePort is a hole in a firewall with a port number stapled to it. What you need is a thing that speaks HTTP — one that reads the Host header, looks at the path, terminates the certificate, and forwards to the right Service. What you need, in other words, is a reverse proxy that Kubernetes knows how to configure — and knows how to keep configured as pods come and go.

Kubernetes does not ship one. It ships the shape of one, twice: the old way (Ingress) and the way that replaced it (Gateway API). This chapter does Gateway API first — not because it is fashionable but because it is where the ecosystem has already moved, and because understanding it makes Ingress’s limits obvious rather than mysterious. Then we install Ingress too, run traffic through it, and give it a fair hearing, because it is what you will find in almost every cluster you inherit this year.

Gateway API is not part of Kubernetes

Start here, because everything else follows from it and almost every tutorial buries it.

Deployment, Service, ConfigMap, Secret — those are built into the API server. They are compiled in. You cannot have a Kubernetes cluster without them. Gateway is not like that. Gateway API is a set of CustomResourceDefinitions plus a controller that watches them, maintained by a Kubernetes SIG but shipped and versioned separately — on its own release cadence, against whatever cluster version you happen to be running.

You install the API by applying a YAML file:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml

That is Gateway API v1.4.0, the standard channel — the graduated, stable-enough-to-depend-on subset. It adds six CRDs to your cluster and nothing else:

  • gatewayclasses — which controller implementation to use
  • gateways — a listener: a port, a protocol, a certificate
  • httproutes — the routing rules themselves
  • grpcroutes — the same, for gRPC
  • referencegrants — permission for a route in one namespace to point at a Service in another
  • backendtlspolicies — how the gateway should speak TLS to your backends

Six new nouns your API server understands — that is the entire installation. There is an experimental channel that adds more (TCP, TLS and UDP routes, among others); I did not install it, so nothing in this chapter depends on it.

Notice what did not happen: nothing started listening on anything. We taught the cluster to store Gateways — we did not give it anything that acts on them. And that distinction is not a pedantic one, because you can now write a perfectly valid Gateway and watch it do absolutely nothing.

A Gateway with no controller is inert

Here is the Gateway for the bookshop. One listener, HTTP, port 80, accepting routes from its own namespace:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: bookshop-gw
spec:
  gatewayClassName: nginx
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: Same

Apply it. The API server accepts it happily — the CRD’s schema validates, the object is stored, kubectl get gateway finds it. And then:

NAME          CLASS   ADDRESS   PROGRAMMED   AGE
bookshop-gw   nginx                          6s

ADDRESS is empty. PROGRAMMED is empty. Not False — empty, because nothing has even had an opinion about this object. And when you go looking for the class it names:

$ kubectl get gatewayclass
No resources found

There is no nginx GatewayClass — there is no GatewayClass at all. We declared that this Gateway should be implemented by a class that does not exist, and Kubernetes said nothing, because saying something is a controller’s job and there is no controller. The object sits in etcd, syntactically impeccable, doing nothing. Nothing is listening on port 80. Nothing ever will.

This is the single most useful thing to internalize about Kubernetes extensions, and it is why this chapter and the chapter on extending Kubernetes are really the same chapter told twice. A custom resource is data. A controller is what makes data mean something. Apply a Gateway with no implementation and you have written a wish. The reconcile loop that turns wishes into running processes has to be installed, and it is not in the box.

Installing an implementation

Gateway API is deliberately a specification, not an implementation. Roughly two dozen projects implement it — NGINX, Envoy Gateway, Cilium, Traefik, HAProxy, the clouds’ own load balancers, and Istio, whose gateways and waypoints are Gateway API resources (which is the whole reason this chapter exists in a Kubernetes book that is also groundwork for a service-mesh one). Pick one. I used NGINX Gateway Fabric — nothing in the chapter depends on that choice, which is rather the point:

helm install ngf oci://ghcr.io/nginx/charts/nginx-gateway-fabric \
  --create-namespace -n nginx-gateway

Now look again:

$ kubectl get gatewayclass
NAME    CONTROLLER                                   ACCEPTED
nginx   gateway.nginx.org/nginx-gateway-controller   True

The controller has installed a GatewayClass named nginx, whose CONTROLLER field is a string — gateway.nginx.org/nginx-gateway-controller — that says, in effect, I answer to this name. And it has marked it ACCEPTED=True, which is a controller writing to a status field to tell you it is watching.

And our Gateway, the one that had been inert for the last few minutes, wakes up on its own. Nobody re-applied it. The controller listed the Gateways that name its class, found one waiting, and reconciled it:

PROGRAMMED   True   ("The Gateway is programmed")

Programmed is the condition that matters, and the word is precise: it does not mean “accepted”, it means the data plane has been configured to serve this. Something is now listening. To prove it, look at what the controller created without being asked — no manifest of mine mentions any of this:

NAME                                     READY   STATUS
pod/bookshop-gw-nginx-5b986c4f69-jxjv8   1/1     Running

NAME                        TYPE           PORT(S)
service/bookshop-gw-nginx   LoadBalancer   80:31564/TCP

That pod is an nginx process; that Service is how traffic reaches it. Neither is in any file I wrote. I wrote a Gateway; a controller read it and materialized a proxy. This is the reconcile loop from the first chapter, wearing a different hat: desired state in, real processes out, forever.

One honest wrinkle, and it is the one that trips people on a laptop. Run kubectl get gateway now and the ADDRESS column still says <pending>. That is not a failure. A LoadBalancer Service asks the cloud for an external IP, and kind is not a cloud — there is nobody to answer. The Service still gets a nodePort (31564), the pod still serves, and routing works perfectly; the only thing missing is the external IP that a cloud provider, or MetalLB, or kind’s own cloud-provider shim would have filled in. PROGRAMMED=True with ADDRESS=<pending> is a working gateway on a cluster with no load balancer, and you should not chase it.

The three roles, and why the split is the point

Look at what we just did and notice that it took three separate objects to get one proxy running. That looks like ceremony. It is not — it is the entire design thesis, and it is what Ingress got wrong.

GatewayClass is the infrastructure provider’s object. Somebody decided that this cluster’s gateways are implemented by NGINX Gateway Fabric. In a real organization that somebody is a platform team, or the cloud vendor, and they set it up once. You will typically never write a GatewayClass; the helm chart wrote ours.

Gateway is the cluster operator’s object. It says: there is a listener on port 80, HTTP, and here are the rules about who may attach routes to it. If it were HTTPS, this is where the certificate would live — which means the person who holds the wildcard cert for *.example.com writes this object, and nobody else needs to see it.

HTTPRoute is the application developer’s object. It says /books goes to my catalog Service. It does not know what a certificate is, cannot allocate a port, and cannot get itself an IP address. It attaches to a Gateway that already exists, and the Gateway’s allowedRoutes decides whether it is allowed to.

Three objects because there are three people — and they have different permissions, different blast radii, and different fears. The operator does not want an app team editing the TLS config. The app team does not want to file a ticket to add a path. Under Ingress, all three concerns live in one object that the app team writes — which is precisely why every Ingress controller grew a pile of annotations, and why cluster operators ended up writing admission webhooks to police what app teams put in their Ingress objects. Gateway API’s answer is a real API boundary instead of a policy about a text field. That’s the redesign. Everything else is detail.

Routing real traffic

Here is the route. Three paths, three Services — all of which have existed since the Services chapter, and none of which change at all to be exposed:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: bookshop
spec:
  parentRefs:
    - name: bookshop-gw
  rules:
    - matches:
        - path: { type: PathPrefix, value: /books }
      backendRefs:
        - name: catalog
          port: 80
    - matches:
        - path: { type: PathPrefix, value: /orders }
      backendRefs:
        - name: orders
          port: 80
    - matches:
        - path: { type: PathPrefix, value: / }
      backendRefs:
        - name: web
          port: 80

parentRefs is the attachment: this route asks to hang off bookshop-gw. The Gateway’s allowedRoutes.namespaces.from: Same lets it, because they are in the same namespace. Had the route been in another namespace, the Gateway would have refused — and the fix would be a ReferenceGrant in the target namespace, which is the object that lets a team say “yes, routes from over there may point at my Service”, rather than a cluster-wide shrug. (I did not exercise a cross-namespace route in the lab; know the mechanism exists.)

Send traffic through the gateway and you get three different applications on one address:

GET /        -> 200   <h1>The Bookshop (SUMMER SALE)</h1>
GET /books   -> 200   {"books":[{"isbn":"978-0134494166",...
GET /orders  -> 200

/ is the storefront’s HTML, right down to the greeting we set from a ConfigMap. /books is the catalog’s raw JSON — not proxied through the storefront, but served by the catalog Deployment directly, because the gateway matched the path and picked a different backend. /orders is the orders service. One port, one hostname, three services — matched by path, by a proxy nobody wrote a config file for.

Two details in that route that reward a second look. Matching is not first-match-wins by document order — the specification is explicit that more specific matches beat less specific ones, so the / rule does not swallow /books even though it appears last and would match every request. You can reorder that YAML however you like and the behavior is the same. And PathPrefix matches on path segments, not on characters: /books matches /books and /books/978-0134494166, and does not match /booksellers. If you have ever debugged an nginx location block that matched a string prefix and routed /healthzzz somewhere surprising, you will appreciate that the segment rule is in the spec rather than in a particular proxy’s config language.

The weighted canary — the thing Ingress cannot express

Now the part that justifies the whole redesign.

You have web running bookshop:v1. You build bookshop:v2, deploy it as web-canary with its own Service and a greeting that makes it obvious which one answered, and you want a small, real slice of production traffic to hit it. Not a copy of traffic, and not a separate hostname you have to convince someone to visit — the actual front door, 10% of it.

In Gateway API that is one field. A rule may have more than one backendRef, and each backendRef may have a weight:

    - matches:
        - path: { type: PathPrefix, value: / }
      backendRefs:
        - name: web
          port: 80
          weight: 90
        - name: web-canary
          port: 80
          weight: 10

That is it. No annotation, no second Ingress object, no controller-specific flag. It is a field in the standard API, and every conformant implementation must honor it.

Run a hundred requests through the gateway and count what came back:

     85 <h1>The Bookshop (SUMMER SALE)
     15 <h1>CANARY BUILD

Eighty-five and fifteen — on a 90/10 weight.

Stop and look at that, because it is the kind of number a tutorial would quietly not print. Weights are probabilistic, not a quota. The gateway is not maintaining a counter and dealing out every tenth request to the canary; it is making an independent weighted choice per request. Over a hundred requests, a 10% probability lands on fifteen about as often as it lands on eight — that is just what binomial noise looks like at n=100. Nothing is broken and nothing needs tuning — the number is the number.

The practical consequence is a rule about how you read a canary. If you send 100 requests and observe 15 hitting the canary, you have learned nothing alarming. If you are computing an error rate for the canary from a few dozen requests, your confidence interval is enormous and you are about to promote or roll back a build based on noise. Weighted routing is a traffic-shaping primitive, not a sampling guarantee — the honest way to say it out loud in a design review is “approximately 10%”, and the honest way to use it is with enough traffic that the approximation stops mattering. And the weights are not percentages, nor do they have to sum to 100. weight: 3 and weight: 1 is a perfectly good 75/25. The implementation divides by the sum.

And a weight of 0 is not “remove this backend” — it is “send this backend no traffic, but keep it as a valid target”. That is the difference between a paused canary and a deleted one, and it is a field you set rather than a block of YAML you comment out.

Promotion, then, is arithmetic on two integers. Nudge 90/10 to 50/50, watch, then 0/100, then delete the old backendRef and repoint the surviving one at web once you have rolled bookshop:v2 into the main Deployment. Every step is an edit to one HTTPRoute, applied in under a second, with no pod restarted anywhere — which is the real difference between traffic-shifting and the rolling update we did in the Deployments chapter. A rolling update moves pods, and whatever traffic split it hands you is incidental to how many of each version happen to be up. Weighted routing leaves both versions running, fully warm, and moves requests — and it rolls back by editing a number rather than by waiting for a fleet of containers to come back up.

Filters: the annotations, promoted to fields

Weights are the headline, but the more quietly important thing in an HTTPRoute rule is the filters list, because it is where the rest of the annotation zoo went. The standard channel defines, among others, a RequestHeaderModifier (set, add, or remove request headers), a ResponseHeaderModifier, a URLRewrite (rewrite the path or the hostname before forwarding — the thing you reach for when your backend expects / but the world calls it /books), a RequestRedirect, and a RequestMirror, which sends a copy of the traffic to a second backend and throws the response away. That last one is how you exercise a new build against production traffic with zero blast radius, and under Ingress it was — you guessed — a controller-specific annotation.

I ran path matching and weights against the lab and printed the results above. I did not exercise the filters, so take the list as a map of the territory rather than as a verified result. What matters for now is the shape of the thing: they are typed fields inside a schema the API server validates, not strings in a map, and a conformant controller that ignores them fails its conformance tests.

TLS, and the certificate that isn’t yours to hold

The Gateway we built listens on port 80 with protocol: HTTP, because on a laptop there is nothing to get a certificate from. In a real cluster the listener grows a protocol: HTTPS, a tls block with mode: Terminate, and a certificateRefs list pointing at a Secret of type kubernetes.io/tls — the one that holds tls.crt and tls.key. cert-manager usually puts it there for you.

I did not stand up a certificate in the lab, and I am flagging that. Two things about the shape of it matter anyway. First, the certificate is referenced from the Gateway, which is the cluster operator’s object — so the wildcard cert for *.example.com never appears in an app team’s HTTPRoute, and an app team cannot accidentally publish a hostname they do not own. That is the role split earning its keep in the one place where a mistake is expensive. Second, this is frontend TLS — the gateway decrypting the world’s traffic. The connection from the gateway onward to your pods is a separate question, and it is what the sixth CRD, BackendTLSPolicy, exists to answer. If you have ever wondered where a service mesh gets its opening, it is exactly there: in the gap between “TLS terminated at the edge” and “plaintext HTTP flying around the cluster”.

Ingress: the legacy path, and a fair hearing

Now install the old thing and run traffic through it, because you are going to inherit clusters full of it and contempt is not a strategy.

helm install ingress-nginx ingress-nginx/ingress-nginx -n ingress-nginx --create-namespace

Same three paths, expressed the way we did it for the last eight years:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: bookshop-legacy
spec:
  ingressClassName: nginx
  rules:
    - http:
        paths:
          - path: /books
            pathType: Prefix
            backend:
              service:
                name: catalog
                port: { number: 80 }
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port: { number: 80 }
$ kubectl get ingress
NAME              CLASS   HOSTS   PORTS
bookshop-legacy   nginx   *       80

And it works. GET / returns 200 and the storefront. GET /books returns 200 and the catalog’s JSON. Path routing, host routing, TLS termination via a tls: block pointing at a Secret — for the job Ingress was designed to do, Ingress does the job. It is stable, it is in every cluster, every engineer has seen one — and if your requirement is “put these three services behind one hostname,” an Ingress object is eleven lines and you are done. Do not rip out a working Ingress because a blog post told you to.

Here is where it stops. Try to express that 90/10 split. There is no field for it — the Ingress schema has host, path, pathType, and a backend Service. That is the whole vocabulary. What ingress-nginx offers instead is this:

metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"

A second Ingress object, identical to the first, marked as a canary by a string key in an annotations map. It works — on ingress-nginx. It is meaningless on Traefik, meaningless on HAProxy, meaningless on your cloud’s load balancer controller — and meaningless on the next controller your platform team migrates to. The API server will happily store it either way, because to the API server an annotation is a string in a map; validation is whatever the controller feels like doing.

That is the whole indictment, and it is not really about canaries. Every capability Ingress lacks — rewrites, timeouts, retries, header matching, session affinity, rate limits, mTLS to the backend, traffic mirroring — got implemented as an annotation, by each controller, with a different name and different semantics. ingress-nginx alone documents well over a hundred of them. Your “portable Kubernetes manifest” acquired a hard dependency on one specific proxy, expressed in a field the API cannot type-check, and you find out on migration day. Gateway API’s answer is not “we invented weights” — it is the weight is a field with a schema, and conformance tests that fail your controller if it ignores it. The API grew a place to put the thing, instead of a place to put a string that a particular controller happens to read.

Ingress is not deprecated and is not going away; it is frozen. The people who maintain it have said, plainly, that new features go to Gateway API — the schema you have is the schema you get. So the sensible position is the boring one: keep the Ingress you have, write new front doors as Gateways, and when you hit the first thing an annotation can’t do portably, you will already be standing in the right place.

Final thoughts

The thing worth carrying out of this chapter is not the YAML. It is the moment where a valid Gateway sat in the cluster with an empty PROGRAMMED column and nothing listened. That is a complete, correct Kubernetes object doing nothing at all, and it is the clearest possible demonstration that the API server is a database with a schema — the behavior you actually want lives in a controller that somebody has to install, that reconciles your declaration into a running nginx process, and that writes back a status field to tell you whether it worked. Read the status. PROGRAMMED is not decoration — it is the only place the cluster tells you whether your front door exists.

And keep the Ingress-versus-Gateway argument in the shape it deserves. Ingress didn’t fail because it was badly designed; it failed because it put three people’s concerns in one object and then let vendors extend it through a map of untyped strings. The result was manifests that only ran on the controller they were written for. Gateway API’s fix was to make the extension points into fields — with a schema, a conformance suite, and a role boundary — and the canary is just the most visible example of a field existing where an annotation used to be. When you meet Istio’s gateways and waypoints later, you will not be learning a new API. You will be attaching routes to a Gateway, exactly as you did here, with a different string in gatewayClassName.

Next: Everything Your Pod Can Do, and Nothing Else

Comments