Nobody Instrumented the Bookshop

One Prometheus query returns the record of every experiment in this book — the fault injection, the timeout, the denials — from an application that has never contained a single line of monitoring code.

The bookshop is about eight hundred lines of Go, written for a Kubernetes tutorial by somebody who was not thinking about observability at all. It imports no metrics library. It has no tracing SDK, no OpenTelemetry collector, no statsd client, no /metrics endpoint. It has not been recompiled since we started this series, it has never had a sidecar injected into it — and it does not know that a service mesh exists.

Here is one Prometheus query against it:

$ sum(istio_requests_total{destination_service_name="catalog"}) by (source_workload,response_code,reporter)

  source=web                reporter=waypoint  code=200   requests=134
  source=orders             reporter=waypoint  code=200   requests=132
  source=orders             reporter=waypoint  code=500   requests=46
  source=web-canary         reporter=waypoint  code=200   requests=7
  source=orders             reporter=waypoint  code=403   requests=2
  source=bookshop-gw-istio  reporter=source    code=200   requests=1
  source=orders             reporter=waypoint  code=504   requests=1

Now read it slowly, because that table is not a demo — that table is the record of this entire book.

The 500s are the fault injection from the resilience chapter — catalog failing half its requests on /debug/fail?pct=50, and 46 of them landing in the metrics as a permanent record of a service that was, for a few minutes, genuinely broken. The 504 is one single request: the one the waypoint gave up on at exactly one second while /debug/slow?ms=3000 sat there sleeping. The 403s are the two requests that the authorization policy denied at L7 once the waypoint was in the path — the DELETE and the POST from the last chapter, counted, labelled, and attributed to their caller. The seven web-canary requests are the canary from the routing chapter, the 10% side of a weighted split, showing up in the metrics as a distinct workload without anybody having to tell Prometheus that a canary existed.

Every experiment in this book is sitting in that query — the failures we injected, the request we timed out, the requests we refused. Nobody added a metrics library. Nobody changed a line of Go. Nobody injected a sidecar.

This is the part of a service mesh that people install it for last and value most in the end — and it is the part I would install one for first.

What is in the metric, and what is not

istio_requests_total is a counter — one per request, incremented by a proxy that saw the request go past. The four labels in that query are the four I actually pulled, and they are the four that carry most of the meaning:

  • source_workload — who called. Not an IP, not a pod name that changes on every rollout. The workload. This is the label that turns a pile of counters into a graph.
  • destination_service_name — what they called.
  • response_code — what happened.
  • reporter — which proxy is telling you this, which turns out to be the most ambient-specific thing on the list, and gets its own section below.

The standard Istio metric carries considerably more than that — source and destination principals (the SPIFFE identities from the mTLS chapter, which means your metrics are authenticated rather than merely labelled), the request protocol, the connection security policy, the destination workload and version. I queried four labels. The rest are documented, they sit on the same time series, and I am telling you about them rather than showing them to you.

That richness has a bill attached, and it is the first thing an operator will hit. Every distinct combination of every label is a separate time series in Prometheus, so the standard metric set multiplies out fast — sources times destinations times response codes times protocols times principals — and on a real mesh with a few hundred workloads that product is where your Prometheus memory goes. This is the mesh’s most common operational complaint, and the answer is the Telemetry object I did not run: it lets you drop dimensions you will never query on. Nobody warns you about this until the day Prometheus is the largest pod in the cluster.

And then the part that is not in the metric at all, which matters just as much. This counter is generated by a proxy that watched a request cross a network. It therefore knows everything about the outside of your service and nothing whatsoever about the inside. It does not know your heap is at 90%. It does not know your connection pool is exhausted, your worker queue is backed up, or that the 200 it just counted carried an empty list of books because a query quietly returned no rows. A mesh is a perfect observer of the space between your services and completely blind everywhere else — and if you internalise one line from this chapter, make it that one, because it is the line that tells you what you still have to instrument yourself.

reporter= is the ambient tell

Look again at the sixth row of that table, because it is the odd one out and it teaches the mechanism:

source=bookshop-gw-istio  reporter=source    code=200   requests=1

Six rows say reporter=waypoint. One says reporter=source. What is that label actually saying?

reporter names which proxy emitted the sample. That is all it is, and once you know it, ambient’s whole telemetry story falls out of it:

  • reporter=waypoint is the ambient-specific value, and it means a waypoint Envoy generated the metric. A waypoint speaks HTTP, so it can tell you methods, paths, and response codes. All the interesting rows above have this value, and that is not a coincidence — the 500s, the 504 and the 403s are all HTTP facts, and they exist in your metrics only because there was something in the path that could read HTTP.
  • reporter=source / reporter=destination are the classic values, meaning the client-side or server-side proxy. The one row above is our ingress gateway, which is also an Envoy, also speaks HTTP, and reports as the client-side proxy for its own outbound call to the catalog.

Which gives the rule in its accurate form, rather than the folk version:

The reporter label tells you which proxy saw the request. Whether you get HTTP codes at all depends on whether that proxy speaks HTTP.

Waypoints and gateways do — they are Envoys. ztunnel does not. And that is not a limitation to be worked around; it is the same fact that produced the connection reset in the last chapter, showing up again in a different costume. ztunnel cannot report a 403 because it cannot speak a 403, and it cannot report a 500 for the same reason: it never parsed a response. What ztunnel reports is TCP.

What ztunnel gives you, waypoint or not

So what does the L4 half of the mesh give you, on its own? More than you would guess. Here is ztunnel’s view of the same namespace, counting connections rather than requests.

One honest caveat before you read it: this namespace had a waypoint by the time I ran the query — you can see it in the table, opening connections like any other workload. I am not going to crop it out to make a cleaner-looking exhibit. What the table shows is that ztunnel counts every connection it handles and labels both ends with a workload identity, and it does that whether or not a waypoint exists. On a mesh with no waypoint at all you would see this same shape, minus one row. I did not run it that way, so I am showing you the one I have.

$ istio_tcp_connections_opened_total  by (source_workload)

  source=shelf-controller     connections=462
  source=orders               connections=191
  source=waypoint             connections=57
  source=unknown              connections=43
  source=web                  connections=16
  source=catalog              connections=11

No response codes. No methods. No paths. But look at what is there: source_workload. Every connection is attributed to a named workload, because ztunnel terminated the mTLS handshake and knows exactly which SPIFFE identity opened it. This is not IP-based guessing that falls apart the moment a pod is rescheduled — it is cryptographically established identity, attached to a counter, for free, on a namespace where no waypoint is required.

Which means: even an L4-only mesh gives you a real service dependency graph. Who talks to whom, at what volume, with the identities verified. That is the artefact every platform team wants, that nobody has, and that is normally reconstructed by asking six teams to fill in a spreadsheet. Enrolling a namespace produces it as a side effect of encrypting the traffic.

And it will tell you things you did not know. Two rows in that table are worth stopping on.

shelf-controller, with 462 connections — by a factor of two and a half, the chattiest workload in the bookshop. It is not web. It is not orders. It is a little custom controller we wrote in the CRDs and controllers chapter of the previous series, which nobody has thought about since, and which is quietly generating more traffic than the storefront. That is not a bug — a controller with a watch and a reconcile loop is supposed to be busy. But it is a fact about your system that no architecture diagram has ever contained, and you got it by labelling a namespace.

unknown, with 43 connections — traffic ztunnel could not attribute to any mesh workload identity. I did not chase it down, and I am not going to speculate about what it was, because the honest and useful thing to say is this: unknown is the row an auditor asks about. Forty-three connections into your namespace, from something the mesh cannot put a name to. Perhaps it is entirely benign. You do not get to assume it is benign — you get to go and find out, and before the mesh you would not have known there was a question to ask.

There is one more use for that graph, and it closes a loop from the last chapter. An ALLOW policy flips a workload to deny-by-default the moment it exists, so every caller you forget to name becomes an outage — which means the hardest part of writing an authorization policy is not the YAML, it is knowing who your callers are. That table is the answer, measured rather than remembered, with identities the mesh verified cryptographically instead of inferring from an IP. Write the allow-list from the telemetry. Then argue with the architecture diagram, and expect to win.

The four golden signals, and the one you will never get for free

The standard SRE framing is latency, traffic, errors, saturation. It is worth walking through them precisely, because the marketing answer is “the mesh gives you all four” and the true answer is better and more useful.

Traffic — free. istio_requests_total at L7, istio_tcp_connections_opened_total at L4. You saw both. No instrumentation, no code, no agreement between teams about metric naming. This one is simply solved.

Errors — free, but only where you have a waypoint. Every one of the 500s, the 504 and the 403s above came from a reporter=waypoint row. An L4-only path gives you a connection that opened and a connection that closed, and a 500 crossing that path looks exactly like a 200. So the error signal has a price, and the price is a waypoint on the Services you care about. That is a perfectly reasonable price. It is not zero, and anyone who tells you it is has not looked at the reporter label.

Latency — free, and I did not run it. Istio’s standard metric set includes istio_request_duration_milliseconds, a histogram, from which you get percentiles per source, per destination, per response code, without touching your application. I queried only the counters in this lab, so I am reporting the existence of the histogram from the documentation rather than from output I can show you. It is the single most valuable thing in the standard metric set, and it is where I would start.

Saturation — never. Not “not yet”, not “with a plugin”, not “in the next release”. A proxy sits on the wire. It cannot see your thread pool, your heap, your goroutine count, your connection pool, your queue depth, or the lock that half your handlers are blocked on. Saturation is the one golden signal a mesh cannot give you at any price, because saturation is a fact about the inside of a process, and a mesh is by construction outside it. (You can approximate it from the outside — rising latency at flat throughput is what saturation looks like through a proxy — but that is an inference drawn from a symptom, not a measurement, and it arrives late.)

That asymmetry is the most useful thing to carry out of twelve chapters. The mesh will tell you, with total reliability and no effort at all, that a service is slow, who it is slow for, and how slow. It will never tell you why. The why lives inside your process — and what the mesh has actually bought you is the right to stop instrumenting the boring 80%, the request counts and error rates and call graph that every team rebuilds badly and by hand, and to spend your entire instrumentation budget on the 20% that only your own code can see.

Kiali, Prometheus, and the things I did not install

Two addons are running in the lab:

kiali        1/1   1     1     2m40s
prometheus   1/1   1     1     2m40s

Prometheus is where every number in this chapter came from. It scrapes the proxies and stores the counters, and it is the actual product — everything else is a view over it.

Kiali is that view. It is a graph UI — services as nodes, observed traffic as edges, health as colour. The important thing to understand about it, and the reason I have spent this chapter in PromQL rather than in a browser, is that Kiali is not a separate data source. It draws its graph from the same istio_requests_total you just queried. If the metric is not there, the edge is not there — so a namespace with no waypoints will not have a rich L7 graph in Kiali, for exactly the reason it has no HTTP codes in Prometheus: nothing in the path was ever asked to read HTTP. Kiali is excellent and I recommend it, and it will never show you something the metrics do not contain. Learn the metrics first and the UI is a convenience. Learn the UI first and you will spend an afternoon wondering why half of it is empty.

And then the honest part, which is that Grafana, Jaeger and Loki were not installed. The lab runs in a 4 GB Docker VM on a laptop, and there was not room. So I am not going to tell you what those dashboards look like, because I did not look at them. What I can tell you accurately is what they are: Istio ships a set of Grafana dashboards that are, once again, queries over the same Prometheus metrics — a mesh-wide view, a per-service view, a per-workload view — and they are worth installing on day one because somebody has already done the PromQL for you. Loki collects logs. Jaeger collects traces, which is the next section, and which is the one place the free lunch runs out.

Two more things I did not run, and you should know it:

  • The Telemetry API object itself. Everything in this chapter is Istio’s default metric set, straight out of the box, with no configuration whatsoever. There is a Telemetry custom resource that lets you add or remove dimensions on the metrics, turn on access logging, and set the trace sampling rate — mesh-wide, per-namespace, or per-workload. I did not apply one. It is the object you will need the moment you want a custom label or a sampling rate that is not the default, and I am pointing at it, not demonstrating it.
  • Distributed tracing, which I did not run because Jaeger is not installed — and which needs more than a footnote.

The context-propagation lie

Here is the claim you will see on every service mesh slide ever produced: “distributed tracing, with no code changes.”

It is not true, and it is the only claim in this chapter that is not.

Here is why. A proxy sees a request arrive at orders. It can generate a trace ID, stamp it into the headers, and report a span for the hop it just witnessed. Excellent. Now orders does its work and calls catalog. That outbound call is a brand-new HTTP request, constructed by your application code, on a connection your application opened. The proxy in the path sees it, and it has no way on earth to know that this outbound request was caused by the inbound one. It is a different request, on a different socket, possibly on a different goroutine, and there is nothing in the packets that connects the two.

Unless your application puts it there. Your code has to copy the trace headers from the inbound request onto every outbound request it makes as a result — traceparent, or the x-b3-* set, or x-request-id, depending on the propagation format you have settled on. That is context propagation. It is a small change, and it must exist in every service in the chain. If one service in the middle drops the headers, the trace does not degrade gracefully — it breaks, at that hop, and everything downstream becomes an orphan with no parent. What you get is a pile of disconnected single-hop spans that look like a trace and are not one.

So: the mesh will generate the spans, apply the sampling, ship them to the collector, and do all the tedious plumbing. What it cannot do is establish causality across a process it cannot see inside — which is the same limit as saturation, wearing a different hat, and it is worth noticing that the two honest gaps in this chapter have exactly the same cause. The mesh knows the wire and nothing but the wire.

Everything else in this chapter genuinely was free. Tracing is not. Series 3 takes it apart properly.

What you can do now

That is twelve chapters. It is worth being concrete about what has actually changed in the reader’s cluster, because it is more than it feels like.

You can encrypt every connection between every service, with mutual authentication, rotating certificates, and per-service identity — and you did it by labelling a namespace, with zero pod restarts and zero code changes. You can write policy that names who may talk to whom using cryptographic identity rather than IP addresses. You can add an L7 proxy to precisely the Services that need one, and write policy about methods and paths there, and pay for it nowhere else. You can shift traffic between versions by weight, route by header, and run a real canary without a deployment tool. You can put a timeout on a service that has never heard of timeouts, retry a flaky dependency, and eject a broken pod that Kubernetes still believes is perfectly healthy. And you can see all of it — the graph, the rates, the codes, the identities — in metrics that already exist, from an application that was never instrumented.

What is still ahead, in “Istio in Practice”: the sidecars you will inherit, and how to migrate off them. Tracing, done properly. Egress, and the traffic that leaves. JWTs and end-user authentication — a different problem from the service identity we did here, and one people routinely confuse with it. Multicluster. Upgrades, across that brutal two-release support window. And debugging a mesh at 3am, which is its own discipline and its own chapter.

Final thoughts

Everything a service mesh does to your traffic is an intervention. The timeout cut a request short. The retry re-sent one. The ejection removed a host. The policy refused a connection. Each of those is the mesh reaching into the path and changing what happens — and every one of them, as I said two chapters ago, makes a failure quieter than it would otherwise have been.

The telemetry is the only part of the mesh that runs the other way. It is the only feature whose entire purpose is to make things louder: to take a fact that was buried inside a network hop between two processes that nobody was watching, and turn it into a number with a name on it. And it is the feature that requires nothing from you at all — no configuration, no code, no decision. It is on. It has been on since chapter four.

Which is why the most common way to be disappointed by a service mesh is to install one for the policy, spend six months arguing about the policy, and never once open Prometheus. That team has bought the expensive half of the product and left the free half in the box. The mTLS was the excuse — the security team wanted encryption in transit, and somebody had to do something about it. What they actually got, and did not notice, was a complete, identity-attributed, cryptographically authenticated map of how their system really behaves — the thing they have been trying to reconstruct from tribal knowledge and stale diagrams for years, delivered as a side effect of a compliance checkbox.

Go and look at what the mesh saw. The bookshop never asked to be observed, was never modified, and does not know it is being watched — and there is a table with 134 requests from web, 46 injected failures, one timeout and two denials in it, which is a more accurate account of what happened to that application in the last hour than the application itself could give you.

That is the trick, and it is worth remembering that it is a trick. It works exactly as far as the network goes, and not one inch further.

Comments