Fast and Cheap Without Getting Dumber: Optimizing a Claude System

There is an order to this work, and optimization is at the end of it. You make the system correct first, then you measure it (chapters 12 and 13), and only then do you make it fast and cheap. The order is not arbitrary. Optimizing before it works is polishing something that might get thrown away. Optimizing before you measure means you cannot tell whether a change helped, hurt, or quietly broke a quality axis you were not watching. The eval from the last two chapters is the safety net under everything in this chapter. Every optimization is a bet that you can spend fewer tokens or less time without losing quality, and the eval is how you collect on the bet or catch the loss.

What optimization is here — and the trap it sets

For a Claude system, optimization means reducing tokens, latency, and cost while holding quality at its target. Those three are the SLA the bookshop platform signs up for: a per-request cost that keeps the assistant cheaper than a human agent, and a response time a customer in a chat window will tolerate. The trap is that every lever trades against quality, and some trade hard. Truncate the context and the answer loses grounding. Downsize the model and the hard cases start failing. The discipline is to pull each lever and then re-run the eval to confirm quality held. An optimization that improves cost and drops your pass rate from 7/7 to 5/7 is not an optimization; it is a regression you chose on purpose without noticing.

So the mental frame is a constrained problem, not a free win: minimize tokens, latency, and cost subject to quality staying above its bar. When there is no eval bar, you are not optimizing, you are just making the system worse and cheaper.

The landscape of levers

An architect has a handful of well-worn levers, and the skill is matching the lever to the constraint you are actually fighting — cost, or latency, or both.

• Output-length constraints. Output tokens are billed at several times the input rate ($5 versus $1 per million on claude-haiku-4-5, for example). They are generated one at a time, so a long answer costs more and takes longer. Ask for exactly the length the task needs. A support answer that must be three sentences should say so in the prompt, and structured extraction should return the fields and nothing else. This is the cheapest lever and the first to reach for.
• Prompt caching. When a large, stable prefix rides on every request — a policy document, a system prompt, a tool list — you pay to process it every single time. Caching lets you pay for it once and reuse it. This is the lever for the exam’s scenario below, and the one worth measuring, so it gets its own section.
• Batching. For work that is not latency-sensitive — an overnight re-embed, a bulk classification, a nightly eval run — the Batch API processes requests asynchronously at half the per-token price. You trade immediacy for cost, so it fits background jobs and never the live chat path.
• Model routing. Not every request needs your most capable model. Route the easy, high-volume cases to a cheaper, faster model and reserve the expensive one for the hard ones — the model-selection decision made dynamic. Done well, this cuts both cost and latency on the bulk of traffic while protecting quality where it matters.
• Reducing tokens. Trim what you send. A retrieval step that stuffs ten chunks into context when three would answer the question is paying for seven chunks of latency and cost on every call. Tighter prompts, fewer and better-retrieved chunks, and dropping stale conversation turns all shrink the input.

None of these is free, and the eval is what tells you which ones you can afford. Reach for the lever that fights your actual constraint: output limits and caching for cost, batching and routing for latency-tolerant volume, context trimming for both.

A word on how they combine, because on a real platform you pull several at once. The levers are mostly independent: caching the policy prefix, capping the answer length, and routing to a cheaper model all stack. Together they can cut a request’s cost several-fold. But two of them interact in a way worth knowing: model routing and caching are both prefix-sensitive, because a cache entry is scoped to a single model. Route the same conversation from one model to another mid-flight and you abandon the cache you were building and pay to write it again on the new model. So route per request on a fresh classification, not mid-conversation, and keep a long cached session on one model. The general rule holds: measure the request’s token accounting before and after, and let the eval confirm quality survived the stack.

The 8,000-token prompt, worked

Here is the exam’s sample question 2, and it is a clean test of whether you reach for the right lever. A support assistant sends an 8,000-token static system prompt plus a policy document on every single request. Both latency and cost are concerns. What do you do?

Notice first what the wrong answers are. Truncating the prompt throws away policy the assistant needs to answer correctly — it trades a cost problem for a quality problem, and the eval would catch the new failures. Blindly downsizing to a smaller model might hold latency but risks the quality the 8,000 tokens were there to provide. And it does nothing about the fact that you are re-sending those tokens every time. Both “fixes” attack the symptom (the request is big) instead of the actual waste (the big part is identical on every request).

The right move follows from that observation. The 8,000 tokens are static — the same bytes on every call — and only the customer’s question changes. That is the exact shape prompt caching is built for. So: order the stable content first (the system prompt and policy, which never change), put the volatile content (the customer’s question) last, and enable prompt caching on the stable prefix. The first request writes the prefix to cache; every request after reuses it, processing only the short question at full price. Caching is a prefix match, so the ordering is not optional — a single changing byte near the front (a timestamp, a per-request id) invalidates everything after it. Stable first, volatile last.

Measuring it, so it is not a hope

Caching is easy to believe in and easy to get wrong silently — a stray datetime.now() in the prefix and your cache-hit rate is zero while the code looks fine. So you measure it. Here is a static prompt in the shape of the exam’s scenario, sent twice, reading the real token accounting off usage:

import anthropic
client = anthropic.Anthropic()
MODEL = "claude-haiku-4-5"

# A large, STABLE system prompt: the brand policy carried on every request.
POLICY = ("Section {n}. Bookshop support policy: confirm the order id before "
          "discussing an order; refunds within 30 days; express shipping is a flat "
          "$12, next business day, US and Canada only; PII retained 7 years; escalate "
          "any dispute over $200 to a human. Never invent a policy not stated here.\n")
SYSTEM_TEXT = "".join(POLICY.format(n=i) for i in range(1, 52))

# how big is the static prefix?
ct = client.messages.count_tokens(model=MODEL, system=SYSTEM_TEXT,
    messages=[{"role": "user", "content": "What is the return window?"}])
print(f"static system prompt: {ct.input_tokens} tokens")

# mark the stable prefix cacheable; the question stays volatile (and last)
system_blocks = [{"type": "text", "text": SYSTEM_TEXT,
                  "cache_control": {"type": "ephemeral"}}]

def ask(q):
    u = client.messages.create(model=MODEL, max_tokens=60, system=system_blocks,
        messages=[{"role": "user", "content": q}]).usage
    return (u.input_tokens, getattr(u, "cache_creation_input_tokens", 0),
            getattr(u, "cache_read_input_tokens", 0))

print("request 1:", ask("What is the return window for books?"))
print("request 2:", ask("How much is express shipping?"))

Run against claude-haiku-4-5, the numbers were unambiguous. The static prompt measured 7,968 tokens. On request 1, cache_creation_input_tokens was 7,957 and cache_read_input_tokens was 0 — the prefix was written to cache, and input_tokens for the request proper was just 13 (the short question). On request 2, cache_creation was 0 and cache_read_input_tokens was 7,957 — the entire policy prefix was served from cache, with only 11 tokens processed fresh.

That second request is the whole argument. The 7,957-token prefix that request 1 paid full price to process, request 2 got for a cache read, which is billed at roughly a tenth of the base input rate. The write costs about 1.25× a normal read, and pays for itself by the second hit. Latency drops too, because the model skips re-processing the cached prefix before it starts answering. Neither truncation nor downsizing would have done this, and both would have cost quality. The winning move spent nothing on quality and cut the repeated work to near zero — which is exactly what an optimization is supposed to do.

Tying it back to the SLA

Optimization is not an aesthetic exercise; it is how the platform meets the promises it made. The bookshop assistant has a cost ceiling and a latency ceiling, and the levers map onto them directly. Caching the 8,000-token policy cuts the per-request cost and the time-to-first-token on the live chat path. Routing easy questions to claude-haiku-4-5 and reserving the expensive model for genuinely hard ones protects both budgets at once. Batching the nightly re-embed and the eval runs keeps that volume off the real-time path and off the full-price meter. And every one of these moves is made against the eval from the last two chapters. You pull the lever, re-run the suite, and keep the change only if quality held. That is the loop: correct, then measured, then fast and cheap, in that order, with the eval standing guard the whole way.

Final thoughts

Optimize last, and only what you have measured. The levers — output-length limits, prompt caching, batching, model routing, and trimming context — each trade against quality, so each one is a bet you settle by re-running the eval. When a large static prompt rides every request, the answer is to order the stable content first and cache it, not to truncate the policy or blindly shrink the model. Measured here, the second request served 7,957 tokens from cache and processed 11 fresh. That is the shape of a real optimization — the same answer, far less work — and it is how the platform keeps its latency and cost promises without getting dumber.

Next: guardrails and failure modes — the first chapter of governance, where you design the system to fail safely rather than fluently.