You Can't Improve What You Don't Measure: Evals for Claude Systems

Every chapter so far has made a design decision — a pattern, a model, a retrieval strategy, a prompt. This chapter is about the instrument that tells you whether any of those decisions was right. Without it you are flying on vibes: the demo looked good, the stakeholder nodded, ship it. Vibes do not scale. A support assistant that answers three sample questions well can be wrong on the fourth, and the fourth is the one a customer asks. The only way to know is to measure, and the discipline of measuring a Claude system is evaluation. This is the arc where the platform stops being something you hope works and becomes something you can prove works, and keep proving as it changes.

What an eval actually is

Strip away the tooling and an eval is three things: a dataset of cases, a way to score the system’s output on each case, and a metric you compute across the whole set and track over time. It is a test suite for a probabilistic system. A unit test asserts f(2) == 4; an eval asserts “on these 200 support questions, at least 95% of the answers are grounded and correct.” The assertion is statistical because the system is, but it is still an assertion you can pass or fail, watch move when you change a prompt, and gate a release on.

That framing matters because it tells you what an eval is not. It is not a benchmark score you quote in a deck. It is not a single golden question you eyeball after each change. It is a standing, re-runnable measurement tied to your system doing your task. Its value is that you can run it a hundred times as you iterate and read the trend.

Why you cannot skip it — and the one time to keep it light

The argument is almost tautological: you cannot optimize what you do not measure. Chapter 13 will A/B two prompts and let the numbers pick a winner, and chapter 14 will trade tokens for latency. Neither move is possible without a score to move. But the sharper reason is that regressions in a Claude system are silent. Change one line of the system prompt and the model may start refusing a category of question, or drift from your policy, and nothing throws an exception. The build is green, the demo still works, and the failure only surfaces when a customer hits it in production. An eval is the thing that catches the regression before the customer does.

There is one honest counterweight. Evaluation has a cost: writing cases, maintaining a rubric, paying for judge calls. For a throwaway prototype or a one-off script that a human reviews anyway, a heavy eval suite is over-engineering. The line is the SLA. The bookshop platform has latency, cost, and quality commitments, handles PII, and runs unattended. Any system with a promise attached needs an eval, because the promise is exactly the thing the eval measures. Reach for a light eval when there is no promise, and a real one the moment there is.

The landscape of metrics

“Is it good?” is not one question. A production Claude system is measured on several axes at once, and an architect names each one and attaches a target to it:

• Quality / accuracy. Is the answer correct, grounded in the source, complete, and on-policy? This is the axis everyone thinks of, and the hardest to score because “correct” for free-form text is a judgment, not a string match.
• Latency. How long until the customer has an answer? A correct answer that takes twelve seconds fails a chat SLA. Latency is a first-class metric, not an afterthought, and chapter 14 optimizes it directly.
• Cost. Tokens per request times requests per day is a real line in a budget. A system that is correct but costs more per resolution than a human agent has failed a different SLA.
• Safety. Does it refuse what it should refuse, avoid harmful or out-of-scope content, and hand off when the stakes demand? A support bot that cheerfully answers a medical or legal question is a safety failure even when the answer is fluent.
• Security. Does it resist prompt injection from retrieved documents or user input, and never leak system-prompt secrets or another customer’s data? On a platform holding PII this is not optional.

The trap the exam sets is treating quality as the only metric. The right instinct is that these axes trade against each other: a bigger model lifts quality and raises latency and cost. The architecture’s job is to hit all the targets, which you can only do if you measure all of them.

The landscape of methodologies

Once you know what to measure, you choose how to score each case. There are three methods, and the skill is matching the method to the axis:

• Programmatic / exact-match. A deterministic check: does the output equal the expected value, match a regex, parse as valid JSON, contain a required order id? Cheap, instant, perfectly repeatable — and only usable when correctness is mechanically checkable. Use it for structured output, format compliance, and any answer with a single right value.
• LLM-as-judge. A second Claude call scores the output against a rubric. This is how you evaluate free-form quality, grounding, tone, and completeness — the things no regex captures. It scales to thousands of cases at a fraction of a human’s cost. It is also the method you must be most careful with, because the judge is itself a fallible model.
• Human review. A person reads the output and grades it. The most trustworthy and least scalable method. Reserve it for what genuinely needs judgment — safety-critical categories, the initial calibration of a rubric, and periodic audits of the automated judge — not for grading every request.

These are layers, not rivals. A mature suite runs programmatic checks on everything cheap to check, an LLM judge on the free-form quality axis, and human review on a sampled slice to keep the other two honest.

Building the dataset

The eval is only as good as its cases, and a good dataset is deliberately built, not scraped at random. Three ingredients:

• Representative cases. The questions customers actually ask, in the proportions they ask them. Return windows, shipping costs, membership pricing — the bulk of the traffic. If the eval passes here, the common path is safe.
• Edge cases. The compound question, the ambiguous one, and above all the out-of-scope one — a question whose correct answer is “I don’t know.” These are where a system fails quietly, so they earn their place in the set even though they are rare in traffic.
• A rubric. For each case, an explicit statement of what a passing answer must contain and must not contain. The rubric is what turns a fuzzy “seems fine” into a decision the judge can make consistently. Write it down; a rubric that lives only in your head is not repeatable.

Running it, with real numbers

Here is the harness that scores the bookshop support assistant. Each case carries a rubric of facts the answer must contain. The judge is a Claude call with a forced structured-output tool, so it must return a clean {passed, score, reason} rather than prose you have to parse.

import anthropic
client = anthropic.Anthropic()
MODEL = "claude-haiku-4-5"

KB = ("Returns: 30 days for a full refund; gift cards non-refundable. "
      "Shipping: express is a flat $12, next business day, US and Canada only. "
      "Privacy: order records retained 7 years. Membership: Prime is $49/year.")

DATASET = [
    {"q": "What is the return window for books?", "must": ["30 days"], "forbid": []},
    {"q": "How much is express shipping?", "must": ["$12", "next business day"], "forbid": []},
    {"q": "How long do you keep order records?", "must": ["7 years"], "forbid": []},
    {"q": "Do you ship to the UK?", "must": ["US", "Canada"], "forbid": []},
    # out-of-scope: the correct answer is to refuse, not invent a rate
    {"q": "What rate does the store credit card charge?", "must": ["don't know"], "forbid": ["%"]},
]

def system_answer(q):
    r = client.messages.create(model=MODEL, max_tokens=120,
        system="Answer ONLY from this knowledge base; if it is not there, say you "
               "don't know and do not guess.\n" + KB,
        messages=[{"role": "user", "content": q}])
    return "".join(b.text for b in r.content if b.type == "text").strip()

JUDGE_TOOL = [{"name": "score", "description": "Score an answer against a rubric.",
    "input_schema": {"type": "object", "properties": {
        "passed": {"type": "boolean"}, "score": {"type": "integer"},
        "reason": {"type": "string"}}, "required": ["passed", "score", "reason"]}}]

def judge(q, answer, must, forbid):
    r = client.messages.create(model=MODEL, max_tokens=250, tools=JUDGE_TOOL,
        tool_choice={"type": "tool", "name": "score"},
        messages=[{"role": "user", "content":
            f"Question: {q}\nAnswer: {answer}\nRubric: must contain all of {must}, "
            f"must contain none of {forbid}. Pass only if both hold. Score 1-5."}])
    return next(b.input for b in r.content if b.type == "tool_use")

results = [judge(c["q"], system_answer(c["q"]), c["must"], c["forbid"]) for c in DATASET]
n = len(results)
print(f"pass {sum(bool(r['passed']) for r in results)}/{n}  "
      f"mean {sum(r['score'] for r in results)/n:.2f}/5")

Run against claude-haiku-4-5, the grounded assistant scored 7/7 passing, mean 5.00/5 on the full seven-case set (the five above plus two more edge cases). That includes the out-of-scope credit-card question, where the assistant correctly answered “I don’t know” rather than inventing a rate. That is the case a system without an explicit grounding instruction tends to fail. A perfect score on seven cases is not proof the system is flawless. It is a small-sample green light that says “no regression here yet,” which is exactly what you want the suite to tell you on every change. The fix for small-sample is more cases, not a different metric.

The judge is a system too

The uncomfortable part of LLM-as-judge is that you have replaced one fallible model with two. A judge can be lenient, inconsistent, or fooled by a confident-sounding wrong answer, and a suite that trusts an unvalidated judge is measuring nothing. So you validate the judge the same way you validate anything: with known-answer cases. Feed it an answer you know is correct and one you know is wrong for the same question, and confirm it separates them.

That validation ran as a side effect of this session. On “Do you ship to the UK?” the grounded assistant answered “US and Canada only” and the judge passed it at 5. A different, ungrounded variant answered “Yes, we do ship to the UK!” — a flat fabrication — and the judge failed it at 2. Same question, opposite truth, and the judge sorted them correctly. That is the minimum bar: a judge that cannot fail a known-bad answer is not a judge. Calibrate it on a handful of known cases, keep a human-reviewed sample to catch drift, and only then trust its scores on the cases whose answers you do not already know.

Final thoughts

An eval is a dataset, a scoring method, and a tracked metric — a test suite for a probabilistic system, and the precondition for every improvement that follows. Measure quality, latency, cost, safety, and security as distinct axes with distinct targets. Score each with the cheapest method that fits: programmatic where correctness is mechanical, an LLM judge for free-form quality, humans for the safety-critical slice and to keep the judge honest. Build the dataset on purpose — representative cases, edge cases, an explicit rubric — and validate the judge before you trust it. The bookshop assistant scored 7/7 today. The point of the harness is that when tomorrow’s prompt change breaks one of those cases, you will see it here first, and not from a customer.

Next: A/B testing and failure diagnosis — using the eval to pick between two designs by the numbers, and to trace a bad answer to its real cause.