Choosing Models for a System, Not a Single Call

The Developer Foundations track taught you to pick a model for a call: name the task’s tolerances, place them on the quality-latency-cost triangle, and choose the tier that fits. That skill is the atom of this chapter. Have it cold before reading on. But an architect almost never designs a call. You design a system of dozens of distinct model interactions: a retrieval answer here, a refund-reasoning step there, a nightly summarization job, a guardrail classifier. Each has its own tolerances, its own volume, its own failure cost. Choosing one model for the whole thing is like buying one vehicle for a courier company. Whatever you pick is wrong for most of the fleet.

So the architect’s version of the question is not “which model” but “which model where, and what does that choice commit me to.” This chapter builds that system-level view. It then lands on three things the Professional exam presses: routing by task, the way a model choice constrains your feature set, and the operational cost of not pinning a version.

What model selection means at system scale

At the level of a single call, model selection is a tradeoff. At the level of a system, it becomes an allocation problem. You have a portfolio of tasks and a menu of models, and the job is to assign each task the cheapest model that clears its quality bar. The unit of decision moves from the request to the task type.

That reframing matters because the tasks in one platform are wildly unlike each other. Answering “what’s your return window?” from a retrieved policy is a shallow, high-volume, latency-sensitive job. Reasoning about whether a $180 refund on a partially-damaged multi-item order is warranted is a low-volume, high-stakes, judgment-heavy job. They are not the same task run at different speeds. They need different intelligence. An architect who sees them as one workload buys the wrong model for at least one of them.

Why it matters, and when the choice barely does

The stakes are asymmetric, which is what makes this worth deliberate thought. Under-provision the hard task and you ship a flagship feature on a model that reasons past its ceiling: wrong refunds, hallucinated titles, escalations that should never have happened. Over-provision the easy task and you pay Opus prices, at Opus latency, to answer a question Haiku would have nailed. Multiply that by every request in your highest-volume path. At scale, the second mistake is quietly the more expensive one, because it compounds per call rather than per incident.

And yet the honest counterweight: for a low-volume internal tool that runs a few hundred times a day, the difference between tiers is rounding error on both the bill and the latency budget. Spending an afternoon routing that system is over-engineering. The discipline of naming tolerances is always worth it. The machinery of per-task routing earns its keep only when volume, cost, or a real quality gap makes the allocation matter. Reach for it when the numbers say so, not because it feels sophisticated.

The landscape: tiers and the triangle

The menu, from the Foundations chapter, is three capability tiers that trade the same three things. Haiku is fastest and cheapest, excellent for classification, extraction, routing, and grounded retrieval answers. Sonnet is the balanced default, strong on most production work at volume pricing. Opus is the most capable, for hard multi-step reasoning and nuanced judgment, at the highest cost and latency. Above Opus, a flagship tier exists for the most demanding long-horizon reasoning, but most platforms never need it.

Every assignment is a point on the quality-latency-cost triangle, and you cannot maximize all three. The architect’s addition to the Foundations picture is that a system holds many such points at once, and they need not agree. The right design routinely runs Haiku, Sonnet, and Opus side by side, each on the tasks whose tolerances match its tier.

Routing by task

The central pattern is routing: send the easy majority to a cheap model, escalate the hard minority to a stronger one. Most production traffic is easy. In a support platform the overwhelming share of turns are FAQ-style questions answered from retrieved policy, and a small tail demands real reasoning. Let a cheap model handle the majority and only the tail escalate, and you get most of the quality of an all-Opus system at a fraction of its cost.

Routing takes two shapes, and both are worth knowing:

• Static routing assigns a model per task type at design time. The refund-reasoning step is always Opus; the retrieval-answer step is always Haiku. Simple, predictable, and easy to reason about, because the mapping is fixed and visible.
• Dynamic routing decides per request, usually with a cheap first pass. A Haiku classifier reads the incoming turn, labels it easy or hard, and forwards the hard ones up a tier. This captures cases where difficulty is a property of the specific request rather than the task type. It also adds a routing hop, its own latency, and a new failure mode: a misroute that sends a hard question to a model that cannot handle it.

The architect’s instinct is to prefer static routing where the task type already predicts difficulty, and to add dynamic routing only where difficulty genuinely varies within a task. Both beat the two lazy defaults the exam offers as distractors. One model everywhere is wrong for most of the fleet. The strongest model everywhere is correct, and needlessly expensive.

A model choice constrains your feature set

One fact separates system design from per-call selection: models are not just faster or slower versions of each other. Their capabilities differ, and picking a model can decide which API features your code may use.

Foundations saw this from the call’s point of view: adaptive thinking is rejected on Haiku with a 400. The architect has to see it from the system’s point of view. Your design depends on certain features, and those features vary across tiers: an extended-thinking mode for a reasoning step, a vision-resolution tier for document understanding, a large context window for a long RAG prompt, structured-output enforcement for a validated tool, a prompt-caching minimum that decides whether your cache even engages. So model selection is not a knob you turn at the end. It is a constraint that flows through the architecture.

The practical consequence: when you route a step to a cheaper model, check that the cheaper model still supports every feature that step relies on. Say you route the refund-reasoning step down to Haiku to save money, but that step needs an extended-thinking mode Haiku does not offer. The code either breaks or silently loses the capability, and the saving is a false economy. Decide the model and the feature set together, per step. Test that the pairing works rather than assuming parity across the lineup.

The cost of a floating alias in production

The last system-level concern is operational. You can name a model by a floating alias that always resolves to the latest snapshot in a tier, or by a pinned dated snapshot that never moves. In development the alias is convenient. In production it is a liability, because a new snapshot can change behavior underneath you with no deploy on your side. Tokenization can shift, moving your token counts, your cost, and your context fit for the same prompt. A prompt tuned to one snapshot’s tendencies can regress on the next. A parameter shape can be deprecated or newly required.

A model upgrade is a breaking change, and it should pass through the same gate as any dependency upgrade: pin the dated snapshot, and re-evaluate before switching. Lock the snapshot in every production path. When a new one ships, run it against your eval set (Arc 4) behind a flag, compare, and promote deliberately. A floating alias skips that gate entirely and hands the decision to whenever the platform happens to roll. For a governed system, that is not acceptable. Changes to production behavior should be evaluated, not stumbled into.

The bookshop platform’s model strategy

Put against our running system, the allocation almost writes itself:

• Retrieval-answer (grounded FAQ over policies, catalogue, and help articles) is the high-volume, latency-sensitive majority. Route it to Haiku. It needs faithful summarization of retrieved text, not deep reasoning, and it runs on nearly every turn — exactly where cheap-and-fast pays off most.
• Refund reasoning (eligibility across order state, damage, partial fulfillment, and the approval threshold) is low-volume and high-stakes. A wrong answer costs real money and trust. Route it to Opus (Sonnet if evals show it clears the bar), and give it the reasoning depth the judgment needs.
• Intent routing at the front door decides whether a turn is a simple lookup or a reasoning case. It is a Haiku classifier, the cheap first pass of dynamic routing that decides whether to escalate.
• The nightly analytics and summarization job over the day’s transcripts is latency-tolerant volume, so it runs on Haiku via batch for the deepest discount.

Every one of those steps is pinned to a dated snapshot, and each pairing is checked against the features that step needs. The refund step’s model must support its reasoning mode; the retrieval step’s must clear the caching minimum from the next chapter. The result is a platform that spends Opus money only on the decisions that earn it, runs the easy majority cheap and fast, and never has its behavior move without an eval saying it may.

Final thoughts

An architect selects models across a system, not for a single call, and the job is allocation: assign each task type the cheapest model that clears its quality bar. Build on the quality-latency-cost triangle, then route by task so the easy majority runs on Haiku and only the hard, high-stakes minority escalates to Opus. Use static routing where the task predicts difficulty, dynamic where it varies within a task. Remember that a model choice constrains your feature set, so decide model and features together per step and test the pairing. And pin every production snapshot, treating a model upgrade as the breaking change it is. Route deliberately and pin ruthlessly, and the platform is both as smart as it needs to be and no more expensive than it has to be.

Next: system prompts and guardrails at scale — designing the trusted instruction layer, the prompt techniques an architect leans on, and where a prompt-level rule stops being enough.