The Log Line Nobody Wrote

Access logs, custom metrics, and trace sampling — turned on for one namespace, with no code change, by an object whose real job is deciding what you are *not* going to pay to collect.

Here is a log line from the bookshop:

[2026-07-14T03:46:09.141Z] "GET /books HTTP/1.1" 200 - via_upstream - "-" 0 680 2 1 "-" "curl/8.14.1" "6054d1a6-91c5-4813-a1fa-84c79b1a45aa" "catalog"

The bookshop does not have a logging library. It has never had a logging library. The Go source has a handful of log.Printf calls in error branches and nothing whatsoever that emits a structured access log, and it has not been recompiled, restarted, redeployed, or edited in any way since we started this book. Nobody wrote that line. Nobody could have — the application does not know the request’s duration in milliseconds, does not know how many bytes went back out on the wire, and has never heard of the UUID sitting in the second-to-last field.

An eight-line YAML object produced it, in one namespace, and it will keep producing one of those for every request through the catalog waypoint until I delete it.

The previous book’s telemetry chapter showed the metrics arriving for free — istio_requests_total with the source workload, the response code and the reporting proxy on it, from an application that has never been instrumented. That chapter was about the fact that the data exists. This one is about controlling it, which turns out to be the harder and more expensive problem, because a mesh’s default answer to “how much telemetry would you like” is “yes”, and on a real cluster that answer has a price.

Decode the line first

Before the object that made it, read the line. It is Envoy’s default access-log format — positional, space-separated, and every field in it is doing a job. Take them in the order they appear:

  • [2026-07-14T03:46:09.141Z] — when the request started, to the millisecond.
  • "GET /books HTTP/1.1" — method, path, protocol. This is the L7 information that only exists because an Envoy read the request. ztunnel could not have given you this and never will; it does not speak HTTP.
  • 200 — the response code.
  • - — response flags, and this is the field you will care most about at 3am. Envoy writes a short code here when it did something to the request: an upstream connection failure, a timeout, a rejected retry, a circuit-break. A - means “nothing to report, the request was ordinary”. A non-- here is the mesh confessing.
  • via_upstream — response code details, and this is a quietly wonderful field. It says the 200 came from the upstream — the catalog actually answered. It is where a manufactured response would announce itself. Which matters enormously given the last chapter: a fault-injected 503 was composed by the waypoint and never touched the catalog, and this field is the one that would tell you so. I did not capture a log line during a fault injection, so I am not going to tell you the exact string it prints — but the field is where you look, and “did my proxy make this response up” is a question you will one day very badly want answered.
  • - and "-" — connection termination details, and upstream transport failure reason. Both empty, because nothing failed.
  • 0 and 680 — bytes received from the client, bytes sent to it. The GET had no body; the book list is 680 bytes.
  • 2 and 1 — total duration, and upstream service time, in milliseconds. Stop here, because these two numbers together are worth the entire chapter.

The two numbers that settle the argument

2 is how long the whole thing took, measured at the waypoint. 1 is how long the upstream — the catalog, the actual application — took. The second line I captured says 8 8, and the third says 1 1.

Subtract, and you have the proxy’s overhead on that request, isolated, per request, in your logs, for free.

This is the field that ends the conversation that begins, roughly eleven days after you install a service mesh, with somebody saying “everything got slower since we turned on Istio”. It is a reasonable thing to suspect. It is occasionally even true. And it is the single hardest claim to argue with, because the person making it has a latency graph that went up and you have a shrug — unless you have these two numbers, in which case you have the request’s total time and the application’s time, at the hop in question, and the difference between them is precisely the thing under dispute. Not an average. Not a benchmark somebody ran on a different cluster. That request, that path, that millisecond.

  • "-" — x-forwarded-for, empty, because the request came from inside the mesh.
  • "curl/8.14.1" — the user-agent. Which tells you, honestly, that the traffic in my lab was me with a shell loop. In yours it will tell you which client library, at which version, is generating the request that keeps timing out.
  • "6054d1a6-91c5-4813-a1fa-84c79b1a45aa" — x-request-id, and this is the most important field in the line.
  • "catalog" — the Host/Authority the request was addressed to.

(The default Envoy format has several more fields after the authority — the upstream host, the upstream cluster, the local and remote addresses. They are not in the lines I captured, so I am decoding what I have rather than what the reference says I might have. Look at your own line before you write a parser against mine.)

x-request-id is the join key

6054d1a6-91c5-4813-a1fa-84c79b1a45aa. The mesh generated it — the application has no idea it exists — and every proxy that touches that request will stamp the same value into its own log line.

Which makes it the thing that turns a pile of per-hop log lines into a request. One grep for that UUID across your log store, and you have every hop the request took, in order, with a duration on each one. That is not a trace, and I am going to be careful about the difference, because it is the entire subject of the next chapter. But it is the substrate a trace is built out of, and it is the same header the mesh injects into the request your application receives, and it is the reason the next chapter starts where it does.

Hold onto that field. It is about to become load-bearing.

The object

Eight lines, in the bookshop namespace, and no code change:

apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: bookshop-logging
  namespace: bookshop
spec:
  accessLogging:
    - providers:
        - name: envoy

That is it — that is the whole thing. envoy is a built-in provider name meaning “Envoy’s own access logger, writing to the proxy’s stdout”, so the log lines land in the waypoint pod’s container output, which means your existing log pipeline — whatever it is — is already collecting them. No sidecar. No agent. No annotation on a Deployment. No restart of anything.

And it applies to one namespace, because the object lives in one namespace. Which is the actual subject of this chapter.

Which proxy is actually writing this

One thing to be precise about, because it decides where you go looking when the lines do not appear.

The three log lines I captured came out of the waypoint — the L7 Envoy bound to the catalog Service, the same one that manufactured the 503s in the last chapter and returned the 403s two chapters before that. They live in that pod’s container stdout. kubectl logs on the waypoint is where you read them — and whatever ships your container logs to your log store is already shipping these, because as far as Kubernetes is concerned they are just a pod writing to stdout like every other pod.

Which means the whole ambient seam applies again, and by now you should be able to predict it: an access log is an HTTP record, so it exists only where something in the path can read HTTP. A Service with no waypoint has no L7 proxy in front of it, so this Telemetry object will produce nothing for it — the same silence, from the same cause, as the unenforced authorization policy and the fault injection that has nothing to inject into. If you apply the object and see no logs, that is the first thing to check, and istioctl analyze is still the fastest way to check it.

What ztunnel does under the same object, I did not look at. ztunnel is L4 and cannot produce a request line — but it does have logs of its own, and it does have a view of connections, and I did not go and read them, so I am not going to characterise them for you. Assume the lines in this chapter are the L7 story, and go and find out what your L4 story is before you build a dashboard on either.

Three levels, and why that is the whole point

A Telemetry object can be attached at exactly three scopes, and the hierarchy is the reason the API exists at all:

  1. The root namespace — istio-system by default. A Telemetry here applies mesh-wide, to everything, in every namespace, forever. This is the platform team’s object.
  2. A namespace — like the one above. It applies to the workloads in that namespace, and it overrides the mesh-wide default for them.
  3. A workload — via a selector on pod labels, or targetRefs pointing at a Service or a Gateway. It applies to that one thing, and it overrides the namespace-level object for it.

Most specific wins. A namespace-level Telemetry beats the root one; a workload-level Telemetry beats the namespace one. (The merge semantics are documented, not measured. I ran the namespace-scoped object and I read the reference for the other two. The Istio docs are explicit that the levels are evaluated in that order and that they do not additively combine — the closest object wins for a given telemetry type. Confirm it on your own mesh before you build a policy that depends on the subtleties, because “the closest object wins” and “the closest object wins for that type” are meaningfully different sentences, and only one of them is going to save you at 3am.)

Now understand why that hierarchy is not a bureaucratic nicety but the load-bearing feature.

You cannot afford access logs on everything. Count the characters in the line at the top of this chapter — it is a little over two hundred bytes, and it is one line, for one request, at one hop. A service handling a thousand requests per second, at two hops each, generates two thousand of those lines per second, which is roughly 400 KB/s, which is about 35 GB a day, from one service. Multiply by the number of services in a real mesh and then look up what your log vendor charges per gigabyte ingested.

That arithmetic is arithmetic, not a measurement — I made up the request rate, and my lab produced three log lines in a curl loop. But the shape is right, and the shape is the point: access logs at 100% across a busy mesh is a line item, not a setting. It is the single most common way a team discovers that observability has a CFO.

So the hierarchy is how you buy the expensive thing precisely where it is worth it. Logs off at the root. Logs on for the one namespace you are currently debugging. Logs on for the one workload that is misbehaving, at full fidelity, while the other four hundred workloads in the mesh cost you nothing. And then — this is the part people forget — you delete the object and the cost goes away, immediately, with no rollout, no redeploy, and no code change, exactly as it arrived.

Turning it off is a feature

The inverse operation is documented and I did not run it, and it is worth knowing about because it is how the hierarchy actually gets used in anger:

apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: quiet-the-health-checker
  namespace: bookshop
spec:
  selector:
    matchLabels:
      app: shelf-controller
  accessLogging:
    - providers:
        - name: envoy
      disabled: true

Logs on for the namespace, off for the one workload that generates ninety percent of the volume and none of the information. The previous book’s telemetry chapter measured shelf-controller at 462 connections — two and a half times the chattiest thing in the shop, a reconcile loop nobody had thought about in months. That is exactly the workload whose logs you do not want to pay to store, and exactly the one that a mesh-wide “turn on access logging” setting would drown you in.

There is a finer instrument too, also documented and also not run — filter.expression, a CEL expression evaluated per request, so you can log only the requests you care about:

  accessLogging:
    - providers:
        - name: envoy
      filter:
        expression: "response.code >= 400"

Errors only. Which, for most production services most of the time, is the correct setting: the 200s are in your metrics already, with percentiles, and they compress to a number. The 4xx and 5xx are the ones where you want the individual request, the user-agent, the path, the response flags, and the request ID you can go and chase. Log what you will read. Count what you will not.

The old way, and why this object exists at all

It is worth a paragraph on what Telemetry replaced, because otherwise its shape looks arbitrary.

Before this API, mesh-level telemetry was configured in meshConfig — a block of YAML in istiod’s ConfigMap. One file. One global setting. accessLogFile: /dev/stdout turned on access logging for the entire mesh, and there was no per-namespace granularity, no per-workload granularity, and no way for the team that owns bookshop to turn logs on for bookshop without an infrastructure ticket to the team that owns the mesh. Changing the sampling rate meant editing the control plane’s configuration and pushing new config to every proxy in the cluster. Every telemetry decision was a platform decision, made in a global file, by a different team.

Telemetry is a Kubernetes custom resource, and everything that follows from that is the improvement. It is namespaced — so RBAC applies, and the bookshop team can be granted the right to create Telemetry objects in bookshop and nowhere else. It is declarative and it lives in the application’s repository next to the Deployment that needs it. It can be created, and deleted, and diffed, and reviewed. And it is scoped, so a decision made for one service does not become a decision made for four hundred.

This is the same trade as the L7 authorization policy, and the same trade as the waypoint: the mesh stopped charging everyone for the thing only some people need, and made you name where you need it. That is the ambient design philosophy applied to observability, and once you see it, you see it everywhere in this project.

meshConfig is not gone — it is still where you declare extension providers, which is how the mesh learns that a telemetry backend exists at all:

meshConfig:
  extensionProviders:
    - name: zipkin
      zipkin:
        service: zipkin.istio-system.svc.cluster.local
        port: 9411

That block is what I applied to point tracing at Jaeger’s Zipkin-compatible endpoint, and it is the one thing in this chapter that is still a global control-plane setting — reasonably enough, since “where does the cluster’s trace collector live” is genuinely a cluster-wide fact. The provider is declared globally and used selectively: meshConfig says the collector exists, and a Telemetry object says which workloads talk to it. That split is exactly right, and it is worth internalising because it is how the tracing configuration in the next chapter is put together.

Sampling, and a number you must not copy

Which brings us to the third thing Telemetry controls, and the one with the sharpest cost gradient:

apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: bookshop-tracing
  namespace: bookshop
spec:
  tracing:
    - providers:
        - name: zipkin
      randomSamplingPercentage: 100.0

One hundred percent. Every request through the bookshop’s waypoints generated a span and shipped it to the collector, for the whole of the tracing work in this book.

That is a lab number and you must not copy it into production. I am saying so here, in the chapter where the knob lives, rather than in the chapter where I use it, because this is the mistake: somebody copies a tutorial’s Telemetry object, ships it, and finds out what a 100%-sampled trace pipeline costs when the bill arrives — or, more entertainingly, when the collector falls over under a load nobody sized it for and takes a chunk of the cluster’s network with it.

I set it to 100 because my lab does about four requests a minute and I needed every trace to exist in order to say anything trustworthy about what the mesh does and does not produce. A trace I did not sample is a trace I cannot reason about, and this book is built on things I actually ran. That is a good reason in a lab and a catastrophic one at scale.

Production sampling rates live in the low single digits of a percent, and often below one — and I am telling you that from the documentation and from experience rather than from a measurement, because I did not run a production mesh at scale to find out. What I can tell you precisely is the mechanism, which is more useful than a number anyway: randomSamplingPercentage is a per-request draw taken at the point the trace begins, and the decision is then propagated downstream in the headers — x-b3-sampled: 1 is the mesh telling every subsequent hop “this one is being recorded, record it too”. So sampling is not “each proxy independently keeps 1% of what it sees”, which would produce a mesh full of traces with holes in them. It is “the edge decides once, and everybody honours the decision” — which only works if the decision is carried, which is a sentence you should read twice, because the next chapter is about what happens when it is not.

Metrics: adding dimensions, and the far more important act of removing them

The last thing Telemetry controls is the metrics themselves, and I did not run any of this. Everything in this section is the documented shape of the API, and I am describing it because you will need it and because the previous book pointed at it and promised this chapter. Treat the YAML as a design to verify, not a result.

You can add a dimension to a standard metric — a tag, computed from a request attribute, that Istio does not ship by default:

spec:
  metrics:
    - providers:
        - name: prometheus
      overrides:
        - match:
            metric: REQUEST_COUNT
          tagOverrides:
            request_host:
              value: "request.host"

And you can remove one, which is the operation that will actually save you:

          tagOverrides:
            source_principal:
              operation: REMOVE

The previous book’s telemetry chapter flagged the bill and did not pay it, so here is the mechanism. Every distinct combination of every label on a Prometheus metric is a separate time series. istio_requests_total ships with a substantial standard label set — source workload, source principal, destination service, destination workload, destination principal, response code, protocol, connection security policy, and more — and the number of series is the product of the cardinalities, not the sum. Sources times destinations times response codes times principals. On a mesh with a few hundred workloads that product is a large number, and it is the number that decides how much memory Prometheus needs, which is why Prometheus has an unnerving habit of becoming the largest pod in the cluster on a mesh nobody tuned.

operation: REMOVE on a dimension you will never group by is the fix, and the honest way to choose which ones is to look at the queries you actually run. If no dashboard, no alert and no incident has ever grouped by source_principal, then source_principal is not observability — it is a multiplier on your memory bill.

There is a disabled: true at the metric level as well, to turn an entire metric off for a scope. Same reasoning, larger hammer, and the same three-level hierarchy decides where it applies.

Final thoughts

The mesh’s observability pitch is that the data is free, and the previous book demonstrated that it is: a table of request counts, response codes, identities and call graph, out of an application that has never been instrumented, at the cost of labelling a namespace. Every word of that is true and I would still install a mesh for that reason alone.

But “the data is free” and “the data is free to keep” are different sentences, and only the first one is on the slide. The generation is free. The collection, the transport, the indexing, the retention and the query are not, and they scale with your traffic rather than with your engineering headcount, which means they scale precisely when you can least afford a surprise. A team that turns on 100% access logging and 100% trace sampling across a mesh on the day of a traffic spike has built themselves a system that gets most expensive at exactly the moment it is under most stress. I have watched that happen. The observability platform fell over before the application did.

So the Telemetry object is not really the “turn on logging” object, even though that is what everyone uses it for first. It is the budget object. Its most important field is disabled, its most important operation is REMOVE, and its most important property is that it is scoped — so that a decision to look closely at one service is a decision about one service, and not an invoice for the entire fleet.

That is a strange thing to say about an observability API, and I mean it seriously. The engineering skill that separates a mesh that helps from a mesh that hurts is not knowing how to switch the telemetry on. Everybody can do that; it is eight lines of YAML, and I showed you them at the top. It is knowing what you are not going to collect — and being able to say why, for each thing, before somebody in finance asks you.

Next: Twenty Traces, One Span Each

Comments