Precision Prompting: Criteria, Not Confidence Hedges

An automated reviewer flags a real bug beside several naming preferences. The developer now has to check every finding before acting on any of them. A useful prompt must define which findings belong in the report and show how to handle cases near that boundary.

Exam objectives covered here. 4.1 Design prompts with explicit criteria to improve precision and reduce false positives. 4.2 Apply few-shot prompting to improve output consistency and quality. Numbering follows the Claude Certified Architect – Foundations Exam Guide, version 1.0, effective July 2026.

What “precision” means, and why it is hard

Precision here is not about writing more, or writing more politely. It is about closing the gap between what you mean and what you said. A model does not share your context or your intuitions; it acts on the words in front of it. When those words are open to interpretation, the model interprets. And it will interpret differently across runs, across inputs, and often not the way you intended. That variance is the enemy. Think of a code reviewer that flags real bugs on Monday and stylistic nitpicks on Tuesday. You don’t trust it, however smart it is on any single run.

“Be conservative,” “use your best judgment,” and “only report high-confidence findings” read like precision controls and deliver none. The model has no calibrated sense of its own confidence to threshold against, so none of them gives the reviewer a checkable definition of an issue. Specify the behavior that warrants a finding and the evidence required to support it.

Explicit criteria beat vague instructions

The foundational lesson is that specific, categorical criteria outperform general instructions. Telling a code reviewer to “check that comments are accurate” is vague: the model has to invent what “accurate” means, and it will flag stylistic quibbles alongside real bugs. Telling it to “flag a comment only when its claimed behavior contradicts the code” is a precise, checkable criterion, and it produces precise findings.

Name the categories to report, such as correctness bugs and security vulnerabilities, and the categories to skip, such as formatting preferences. This gives the model a decision rule and gives you something to evaluate when the output is noisy.

Precision matters because of a trust dynamic: a high-false-positive category poisons the accurate ones. If a reviewer’s “style” findings are usually wrong, developers stop trusting all of its findings, including the correct security ones. The noise trains them to dismiss the tool. So a category that fires too many false positives isn’t just locally bad; it undermines the whole system’s credibility.

That gives a concrete remediation: temporarily disable a high-false-positive category while you improve its criteria, to restore trust in the categories that work. Turning off the noisy check is not giving up. It is protecting the signal. Measure whether the revision makes the removed category worth restoring.

A severity rubric, with a code example at every level

For classification the same principle takes a specific form: define each severity level, and anchor it to a concrete code example. This is the part most teams skip, and it is the part that does the work. “High” versus “medium” is a word until an example pins it down. Without the anchor, the model’s own shifting sense of drama decides, and your severity distribution drifts between runs and between file types.

Before the levels, one decision about what the scale is even measuring. A finding has at least three properties that people routinely fold into a single word. How bad it is if the code runs, how likely anything is to reach it, and how far the damage spreads. Fold those together and the rubric stops being reproducible. Two reviewers who weight them differently will assign different levels to the same defect, and both will be following the rubric.

So the scale below measures impact if the code runs, and nothing else. Reachability and blast radius are recorded as separate fields on the same finding. That is not extra bureaucracy; it is what makes the output composable. A model looking at one file often cannot know whether a function is reachable. That is the cross-file pass’s job in chapter 11. A rubric that demands it in the severity call is asking for a guess dressed as a level. So keep the dimensions apart, let each pass fill in the ones it can actually see, and compute the queue order in your own code, where the weighting is visible and testable.

Here is the rubric written out, in the form you would paste into a review prompt. Each level names a checkable condition and shows a snippet that meets it.

SEVERITY RUBRIC — assign exactly one level per finding.
SEVERITY measures IMPACT IF THIS CODE RUNS. It does not measure how likely
the code is to run. Record that separately in REACHABILITY, below.

CRITICAL — the code as written can lose data, expose data, or execute
attacker-controlled input. Judge the code in front of you; if you cannot
tell whether anything calls it, that goes in REACHABILITY, not here.
    query = f"SELECT * FROM users WHERE email = '{email}'"
    db.execute(query)

HIGH — the code produces a wrong result, or crashes, on an input the
function's own signature permits. The bug is reachable from valid input.
    def average(values: list[float]) -> float:
        return sum(values) / len(values)      # ZeroDivisionError on []

MEDIUM — correct today, but a documented contract is violated or a
resource is not released. Failure requires a second condition to occur.
    conn = pool.acquire()
    rows = conn.query(sql)                    # no release on exception
    pool.release(conn)
    return rows

LOW — behavior is correct; the issue is clarity, naming, or a stale
comment that contradicts the code beneath it.
    # returns the user's full name
    def get_user(uid): return db.users.find_one({"_id": uid})

DO NOT REPORT — formatting, import ordering, or a local convention that
differs from your preference but is used consistently in this file.

Then, on every finding you do report, also record:

REACHABILITY — one of:
    untrusted-input   a caller can drive this with data from outside
    valid-input       reachable from inputs the signature permits
    internal-only     only reachable from code in this repository
    tests-only        the only callers found are tests
    unknown           you cannot tell from what you were shown

BLAST_RADIUS — one of: one-request | one-tenant | all-tenants | stored-data

Three properties make this rubric better than a list of adjectives. Each level states a condition you can check against the code, not a feeling. The levels are ordered by a single axis, so a finding that could plausibly sit at two levels has a tiebreak. And the last block is as important as the first four: naming what not to report is what actually suppresses the noisy category. A rubric with four levels and no exclusion list quietly invites the model to fit every observation into LOW.

The two extra fields are what keep that single axis honest. unknown is a first-class answer rather than a failure, and it is the correct answer from a pass that was shown one file. A CRITICAL finding with reachability tests-only and a CRITICAL finding with reachability untrusted-input are the same defect at wildly different priorities. Your triage code can now tell them apart. Set the weighting yourself and write it down. Then check it against a calibration set — twenty findings whose real priority you already agree on — and look at where reviewers or runs disagree. Disagreement almost always localises to one under-specified level, which you then anchor with another example. A rubric nobody has calibrated is a rubric whose severity distribution you are about to discover in production.

The rubric also composes with the trust dynamic above. When LOW turns out to be your high-false-positive category, you delete that block and the two lines describing it. The remaining levels keep working, and the reviewers who had started ignoring the tool get a clean signal back.

Structure: system prompts and XML tags

Precision is also a layout problem. Two structural levers carry it, and the first is the one that changes most designs.

The system prompt is a separate parameter, and it accepts a list. The signature is system: Union[str, Iterable[TextBlockParam]]. The string form is what most code uses. The list form matters because each TextBlockParam carries its own optional cache_control. That is what makes the system turn the natural home for the long, stable part of your prompt. The SDK is explicit that this is not a message role:

Note that if you want to include a system prompt, you can use the top-level system parameter — there is no "system" role for input messages in the Messages API.

Put the role, rubric and criteria in the system turn, and the material being judged in the user turn. This separates the stable prefix from the changing input; the caching section below shows how to measure reuse.

XML tags are Anthropic’s documented convention for separating the parts of a prompt. Wrapping the instructions, the examples, and the input in named tags gives the model unambiguous boundaries, and gives you a stable place to interpolate:

<criteria>
{the severity rubric}
</criteria>

<examples>
{two or three worked findings, in the exact output format}
</examples>

<code_under_review>
{the diff}
</code_under_review>

Review the code in <code_under_review> against <criteria>. Match the
output format shown in <examples> exactly. Output nothing else.

The tags are a prompt-authoring convention, not an API feature. Nothing in the SDK parses them, and the effect on output quality is a claim about model behavior that this chapter did not measure. What is unambiguous is the engineering benefit on your side: a tagged prompt has named slots, so the template has one obvious place to interpolate each part, and a reviewer can see at a glance which region is data.

Tags do not enforce a trust boundary, and this is the most over-claimed thing in prompt engineering. The model sees one flat sequence of tokens. Nothing enforces that content placed inside <code_under_review> stays inside it, and nothing escapes your interpolated string. A diff containing the literal text </code_under_review> can close its own tag and carry on as though it were your instructions. A diff containing “Review the code” is less likely to be read as an instruction when it sits inside a named region, and “less likely” is the honest strength of the claim; the effect on resistance to injection was not measured here.

Enforce access and output requirements outside the prompt: offer only the needed tools, constrain the response shape, and validate it against the source. Include adversarial inputs in the evaluation set. Chapter 10 develops those checks for incoming documents.

Those two are the portable levers. There used to be a third, and it is worth knowing why it is gone.

A compatibility note: assistant prefill

Older Messages API guidance told you to end the messages list with an assistant turn, so the reply would continue from text you had already written. The SDK reference still describes the behavior:

If the final message uses the assistant role, the response content will continue immediately from the content in that message. This can be used to constrain part of the model’s response.

Do not build on it. Ending a request with {"role": "assistant", "content": "SEVERITY: "} returns a 400 on the current models, not a constrained reply. claude-sonnet-5 and claude-opus-5 reject an assistant final message outright, with This model does not support assistant message prefill. The conversation must end with a user message. That exact string fails on claude-haiku-4-5 too, though for a different reason: a final assistant message may not end in whitespace, which is its own undocumented-looking rule and applies on every model.

Drop the trailing space and the two failures separate:

model	"SEVERITY: "	"SEVERITY:"
claude-haiku-4-5	400 trailing whitespace	accepted
claude-sonnet-5	400 trailing whitespace	400 prefill unsupported
claude-opus-5	400 trailing whitespace	400 prefill unsupported

The measured compatibility differs by model: whitespace-free prefill worked on Haiku and failed on Sonnet and Opus. Test the requested model explicitly instead of assuming a successful Haiku run covers another deployment.

The job prefill used to do belongs to chapter 10. Prefill only ever constrained the start of the output anyway. Structured output through a forced tool call constrains the whole shape, and unlike prefill it is portable. Both forms of forced tool_choice, {"type": "any"} and {"type": "tool", "name": ...}, were verified working on claude-haiku-4-5, claude-sonnet-5 and claude-opus-5. All three returned stop_reason == "tool_use". Reach for that instead.

Few-shot: teaching judgment, not matching

Explicit criteria take you a long way, but they hit a wall: you cannot spell out every case. Where the criteria run out, the model improvises, usually in the format of its answer, or in how it handles a genuinely ambiguous input. When that happens, few-shot examples are the most effective next technique. You show the model 2–4 worked examples of exactly what you want, and it patterns its output on them.

Few-shot examples do more than give the model cases to match. They let it generalize its judgment to novel patterns: a handful of well-chosen examples teaches the principle behind your choices, which it then applies to inputs you never demonstrated. That is why few-shot beats an exhaustive rule list. You cannot enumerate every case, but a few examples that reveal the underlying judgment cover the ones you did not write down. So choose examples that show how to decide a new case; a useful pair shows why two similar inputs receive different outputs. Test that distinction on inputs excluded from the prompt.

The highest-value examples are the ambiguous ones, shown with the criterion that decided them:

Example — ambiguous tool selection:
Request: "What did we spend on cloud last quarter?"
Criterion: the request names a figure over a bounded period, so it is a
lookup, not a document search. Search returns docs *about* cloud spend;
billing returns the number.
Chosen: get_billing(category="cloud", period="last_quarter")

Note what that field is and is not. It is a short written rationale you authored, in the example, stating the rule that separates two plausible actions. It is not a request for the model to narrate its own thinking, and it should not be labeled in a way that invites one. Word the field as “Reasoning” and you drift towards asking the model to disclose an internal process it has no privileged access to — a story about a decision rather than the criterion behind it. Ask for the criterion, keep it to a line, and tie it to something checkable. Where you genuinely want more reasoning before the answer, the mechanism is the thinking parameter later in this chapter. That is a separate feature with its own budget and its own content blocks.

Showing why one action was chosen over a plausible alternative teaches the model to make the same distinction on a new ambiguous request. A rule (“use billing for financial questions”) can’t do that when the phrasing is unexpected. This pays off in several places. It demonstrates a specific output format — location, issue, severity, suggested fix — so results stay consistent. It distinguishes acceptable code patterns from genuine issues, cutting false positives while still generalizing. And it shows correct extraction from varied document structures, such as inline citations versus bibliographies, so the model handles formats it wasn’t explicitly shown.

On a small ticket-classification task run against claude-haiku-4-5, a zero-shot prompt asking for severity and category produced 0 of 4 responses in the required one-line format. The same four tickets with a two-example prefix produced 4 of 4.

This is a small demonstration: four items, one run per arm, one model, and one task. The zero-shot result may reflect an underspecified format instruction. It supports “the examples helped in this run,” without establishing a general improvement rate.

The measurement that would support a general claim is a different exercise, and it is worth knowing what it costs before you cite anyone else’s number. A held-out set in the dozens to hundreds. The sampling parameters pinned and stated. Several runs per arm, because the default temperature, which the knobs section below introduces, is 1.0. An interval around each rate rather than a point. And a look at which items failed, not only how many. That is the eval lifecycle the last section of this chapter describes, and it is the difference between “few-shot helped here” and “few-shot helps.” The claim that few-shot examples also reduce hallucination is documented model behavior rather than anything measured here.

How the examples get into the request

There are three encodings, and they are not interchangeable.

Inline, in one user turn. Everything — instructions, examples, input — goes into a single user message, usually delimited with the XML tags above. Simplest, and the right default. The examples are plainly labeled as examples rather than as history.

As alternating turns. You append real user/assistant pairs to messages, so each example looks like a turn the model already took. This is the strongest format signal available, because the model is completing a conversation in which it has already answered three times in your format. It comes with a trap the SDK states plainly:

Consecutive user or assistant turns in your request will be combined into a single turn.

So a list of five user messages holding five examples is not five turns. It is one turn with five examples concatenated, and you have paid for the alternating-turn encoding without getting it. The pairs must actually alternate.

In a dedicated block. A single <examples> section, usually in the system prompt rather than the user turn, when the examples are stable across every call. This is the encoding that caches well.

Choosing and ordering the examples

Four rules survive contact with real prompts:

• Cover the decision boundary, not the easy middle. Two examples of an obvious CRITICAL teach almost nothing. One example that is HIGH and one that looks nearly identical but is MEDIUM teach the line between them.
• Include a counter-example. An input that a reasonable model would flag, shown with the answer “do not report,” is often worth more than three positive examples. It is the only way to demonstrate the exclusion block of a rubric.
• Order matters, and recency is real. Examples nearest the input carry more weight. Put the case most like your hard inputs last.
• Watch for overfitting to the format. If every example happens to be a Python function, expect worse behavior on the TypeScript file. If every example output has exactly three findings, expect three findings. Vary the incidental features that you do not want copied.

Tools carry their own examples

Few-shot is not only a prose technique. A tool definition has a field for it. In anthropic 0.120.0, ToolParam has these keys:

['allowed_callers', 'cache_control', 'defer_loading', 'description',
 'eager_input_streaming', 'input_examples', 'input_schema', 'name',
 'strict', 'type']

input_examples is typed Iterable[Dict[str, object]]: a list of example inputs for the tool, attached to the tool rather than to the conversation. So the few-shot signal for “what a well-formed call to this tool looks like” belongs next to the schema, not in a system prompt that a later refactor might trim. strict is covered in chapter 10; the caching section below explains cache_control.

The knobs that are not in the prompt

Everything so far treats variance as a wording problem. Some of it is not. The Messages API exposes four sampling parameters that bound how much the model is allowed to vary at all. A prompt-engineering chapter that never mentions them leaves the reader tuning the wrong layer. In anthropic 0.120.0 they sit on messages.create alongside messages and model:

temperature: float
top_p: float
top_k: int
stop_sequences: SequenceNotStr[str]

temperature is the one that matters for precision work. Its documented behavior, from the SDK’s own parameter reference:

Defaults to 1.0. Ranges from 0.0 to 1.0. Use temperature closer to 0.0 for analytical / multiple choice, and closer to 1.0 for creative and generative tasks. Note that even with temperature of 0.0, the results will not be fully deterministic.

Where that parameter lives depends on which SDK generation you are on, and this is the sharpest version seam in the book. On the pinned anthropic 0.120.0, temperature, top_p and top_k are ordinary keyword arguments to messages.create. On anthropic 1.3.0 they are gone from the signature, and passing one raises TypeError: Messages.create() got an unexpected keyword argument 'temperature' — from the client, before any request is sent.

The API did not drop them. The same value delivered through extra_body is accepted and honoured:

# anthropic 1.x: the typed parameter is gone, the wire field is not
client.messages.create(model=MODEL, max_tokens=8, messages=msgs,
                       extra_body={"temperature": 0})

Check the installed package version before copying either form. A removed keyword fails at the call site; extra_body sends fields outside the typed signature, so validate those requests explicitly.

Read both halves. The default is 1.0, the creative end. So an extraction or classification prompt that never sets temperature is running at the setting the documentation recommends for generative work. That is a plausible cause of the run-to-run variance this chapter opened with, and it is a one-line fix. But the second sentence forecloses the obvious next thought: temperature=0 is not a determinism guarantee. It narrows the distribution, it does not collapse it. Any design that requires byte-identical output across two calls needs a cache or a stored result, not a temperature setting.

top_p (nucleus sampling) and top_k truncate the candidate distribution rather than reshaping it. The SDK marks both “Recommended for advanced use cases only.” The practical guidance follows from that: adjust temperature or top_p, not both at once. Move both and you cannot attribute a change to either.

stop_sequences is the one people forget is a precision tool. It hard-stops generation on a literal string, and the model reports it back:

If the model encounters one of the custom sequences, the response stop_reason value will be "stop_sequence" and the response stop_sequence value will contain the matched stop sequence.

Be precise about what that buys, because it is easy to over-read. A stop sequence is a truncation rule, not a grammar. stop_sequences=["\n\n"] guarantees that no second paragraph is generated; it guarantees nothing about the first one being complete, well-formed, or present. Three failure modes follow directly, and all three return HTTP 200:

• The delimiter appears inside a legitimate answer. A finding whose suggested fix contains a blank line stops halfway through the fix. You get a valid-looking string that has silently lost its tail.
• The delimiter never appears. Generation runs to max_tokens instead, and now you are reading a fragment for a different reason. stop_reason distinguishes the two, which is the whole argument for checking it.
• The delimiter is not where your parser expects it. Which sequence matched is reported out of band, on Message.stop_sequence — “which custom stop sequence was generated, if any” — rather than being something you dig back out of the text. Read it from the field; do not assume the content block ends with it.

So use stop sequences the way you would use a length cap: as a bound on runaway generation, with an explicit parse-and-reject step behind it. stop_reason == "stop_sequence" tells you the guard fired, and chapter 1’s stop-reason table is where that value belongs in your loop. But what it carries is “this output was cut,” not “this output is valid.” For anything a program has to consume, the mechanism is structured output, not a delimiter — chapter 10 makes that case in full. A stop sequence keeps a chatty model from writing three paragraphs where you wanted one. It does not give you a machine contract. And a format test that skips the embedded delimiter, the missing delimiter and the max_tokens cut has not tested the thing that will break.

Criteria and examples shape what the model says; sampling parameters bound how much it varies while saying it. They are not substitutes. A vague prompt at temperature=0 gives you the same vague answer reliably.

Few-shot blocks are the canonical thing to cache

A good few-shot prefix is long, and it is identical on every call. That is the exact profile prompt caching exists for, and the API exposes it as a marker you place on a content block:

class CacheControlEphemeralParam(TypedDict, total=False):
    type: Required[Literal["ephemeral"]]
    ttl: Literal["5m", "1h"]
    """Defaults to `5m`."""

The marker can go on a TextBlockParam, including one inside the system list, or on a ToolParam. You can also let the API place it for you: messages.create takes a top-level cache_control whose documented behavior is:

Top-level cache control automatically applies a cache_control marker to the last cacheable block in the request.

Which is why the layout advice above is load-bearing. Cache hits require an identical prefix, so the stable material has to come first. Rubric and examples in the system prompt, the diff or the ticket in the user turn, and the boundary between them is the cache boundary. Reverse that order and you will get a cache write on every single call.

You can see whether it worked, because the response reports it. Usage carries cache_creation_input_tokens and cache_read_input_tokens as separate counters, plus a cache_creation breakdown split by TTL (ephemeral_5m_input_tokens and ephemeral_1h_input_tokens). A first call writes; subsequent calls within the TTL read. There is also a deliberate pre-warm path for filling the cache before a batch starts — max_tokens is documented as “Set to 0 to populate the prompt cache without generating a response.”

Here is a cached prefix working, on claude-haiku-4-5. The same system block, sent twice:

prefix ~  6013 tok   call1(in=   11  write= 6002  read=    0)   call2(in=   11  write=    0  read= 6002)

Note what input_tokens does. It reads 11, not 6,013. The cached prefix moves out of input_tokens and into the cache counters entirely, so a naive cost estimate built on input_tokens alone will look implausibly cheap and tell you nothing about whether the cache is being written every turn.

The minimum, and what happens below it

cache_control is a request to cache, not a guarantee. Below a minimum prefix length the API accepts the marker and ignores it. Bisecting that boundary on this model:

prefix ~  2013 tok   ->  silently skipped
prefix ~  4059 tok   ->  silently skipped
prefix ~  4109 tok   ->  CACHED
prefix ~  6013 tok   ->  CACHED

The threshold sits between 4,059 and 4,109 tokens, which is 4,096 — and that number is per-model, so check it for whichever model you deploy. What matters more than the constant is the failure mode. There is no error, no warning, and no field that says “your cache_control did nothing”. Both counters simply stay at zero, and the only way to know is to look at them. A prompt sitting just under the line is the expensive case: you wrote the caching code, you believe the prefix is cached, and you pay full price on every call.

Read the counters the right way round, because the arithmetic depends on it. Both are per-request, not running totals. cache_creation_input_tokens is what this call wrote; cache_read_input_tokens is what this call read; input_tokens is what neither covered. They partition the prompt for that one request, which is why the three add up to the prefix and why input_tokens collapsed to 11 above. A dashboard that sums them across a day gives you real totals; a single response never carries one.

What invalidates a hit is equally mechanical: the cache key is the exact prefix up to and including the marked block. Change a character of the system prompt, reorder two examples, add a tool definition ahead of the marker, switch models, or simply let the TTL run out. Any of those makes the next call a write rather than a read. That is the whole list, and it is why the stable material has to be genuinely stable. A timestamp or a request id anywhere in the prefix turns every call into a cache write — at a higher price than not caching at all — and the counters are the only place that shows.

What is not claimed here: no cost or latency figure was measured for this book. Treat “caching makes the few-shot prefix cheap” as the vendor’s pricing model rather than a number from these pages. Take the write premium and the read discount from the current price list, not from a book. The structural claim is the verified one. The counters exist, they are separate, and they are how you confirm your prefix is actually stable instead of merely looking stable.

Extended thinking is a budget, not a precision control

One more parameter deserves naming here, because chapter 11 will dismiss it in one clause and a reader should know what is being dismissed. messages.create takes a thinking parameter, and it is a real feature:

ThinkingConfigParam = Union[ThinkingConfigEnabledParam,
                            ThinkingConfigDisabledParam,
                            ThinkingConfigAdaptiveParam]

ThinkingConfigEnabledParam requires budget_tokens, documented as “Must be ≥1024 and less than max_tokens,” and takes an optional display of "summarized" or "omitted". When it is on, responses include thinking content blocks — ThinkingBlock, or RedactedThinkingBlock where the content is withheld. Under streaming they arrive as ThinkingDelta. ThinkingConfigAdaptiveParam carries no budget at all; the model decides how much reasoning to spend.

Which of the two a model accepts is a property of that model, not a setting you choose once. The SDK warns on some models that type="enabled" is being superseded:

Using Claude with … thinking.type=enabled is deprecated. Use thinking.type=adaptive instead which results in better model performance in our testing.

Read as a general rule, that notice is wrong in both directions. Here is what the three current models actually accept:

Model	{"type": "enabled", "budget_tokens": N}	{"type": "adaptive"}
claude-haiku-4-5	works, blocks ['thinking', 'text']	400 adaptive thinking is not supported on this model
claude-sonnet-5	400	works
claude-opus-5	400	works

The deprecation notice is scoped to the models that have moved on, and Haiku 4.5 is not one of them. So write the thinking config per model rather than once for the whole codebase, and treat a model swap as a change that has to revisit it.

The consolation is that getting it wrong is loud. Send the form a model does not accept and the request fails with a 400, so the mistake shows up the first time you run it. There is no path where thinking is quietly ignored and you pay for reasoning that never happened — unlike the caching threshold two sections up, which fails in exactly that way.

Note what the parameter buys: more tokens of internal reasoning before the answer. That helps on problems whose difficulty is depth of reasoning. It does nothing about a criterion you never wrote down. And it does nothing about the shared-context bias chapter 11 describes, because thinking harder inside the same context does not give you a second perspective on it. Whether a given budget improves a given task is a model-behavior claim, and nothing in this book measured one.

A prompt is code, so version it and test it

Everything so far is a way to write a better prompt. None of it tells you whether the prompt you wrote is better than the one it replaced, and that gap is where most prompt work quietly goes wrong. A prompt has all the properties that make software need regression tests. It is edited by several people, its behavior changes when a dependency changes, and it fails silently rather than loudly.

Worse, it has a dependency that moves on its own. A prompt tuned against claude-haiku-4-5 is tuned against that model’s habits, and the model is not yours. Swap it for a stronger one and your carefully-worded workaround for a Haiku quirk becomes dead weight, or worse, an instruction that now fights the model. Three inputs move independently: the prompt, the model, and the tools the prompt describes. A change to any of the three is a change to the system.

Five practices cover it, and none of them require a platform.

• Version the prompt as an artifact, not as a string literal. Give it an id and a version, store it beside the code, and stamp both onto every record it produces. Chapter 10 puts a schema_version on the extraction; the prompt version belongs on the same row. Without it you cannot answer “which prompt produced this output,” which is the first question anyone asks about a bad result.
• Keep a golden set and a holdout set. The golden set is the cases you tuned against, and it will flatter you. The holdout is cases you have never looked at, scored only when you are about to ship. A prompt that improves on the golden set and not on the holdout has been fitted to its examples. That is the prompt-engineering form of overfitting, and just as easy to do by accident.
• Classify the failures, don’t just count them. A pass rate that moves from 82 to 86 tells you almost nothing. A taxonomy — wrong format, missing field, wrong severity, hallucinated value, refusal — tells you what to fix. It also keeps a regression visible when the aggregate stays flat, because one category improved while another got worse.
• Gate the release on the eval, and record the delta. Every prompt edit runs the suite; the result goes in the pull request. Run each arm several times, since temperature defaults to 1.0 and a four-point move on one run is usually noise. If a change cannot be shown to help, it does not ship, however obviously better it reads.
• Re-run it on the schedule, not just on edits. Nothing in your repository changes when the model behind an alias does. A weekly run against the holdout set is how you find out that your prompt drifted, rather than hearing it from the people who use the output.

Keep the evaluation runnable as the prompt changes. A revision is easier to assess when you can compare its failures with those of the version it replaces.

Precision plus generalization

Criteria alone can be rigid and miss novel-but-valid cases; examples alone can be imitated too literally, without the underlying rule. Together they give output that is both precise and able to generalize: a clear criterion, plus a few examples showing the reasoning at its boundary. Build the review prompt that way, keep the code under review in its own clearly marked region, choose sampling settings deliberately, and score the result against cases you did not use to write the prompt.

If the reviewer keeps reporting harmless style differences, inspect that category’s exclusions and counter-examples. If the criterion is clear but results vary, test the model and sampling configuration. Choose the next change from the observed failure rather than adding another general instruction to be careful.

Final thoughts

What to carry forward.

• Precision comes from specific categorical criteria, not from hedges the model cannot calibrate. A noisy category poisons trust in the accurate ones.
• Anchor every severity level to a code example, and keep severity, reachability and blast radius as separate fields.
• temperature defaults to 1.0 and even 0.0 is not fully deterministic. stop_sequences bound generation and are not a machine contract.
• XML tags organise a prompt and do not police it; the trust boundary is the tools, the schema and your validators. Put the stable half in the system turn where cache_control can hold it.
• Few-shot teaches the model to generalize. The best examples show the reasoning at an ambiguous boundary, encoded turns must alternate, and a tool carries its own through input_examples.
• Version the prompt and score it against a holdout set whenever the prompt, the model or the tools change.

Take the reviewer’s last ten disputed findings and classify why they were unhelpful. An unclear severity boundary needs an example. A category full of preference-based comments needs a sharper exclusion. An unsupported claim needs a requirement for evidence.

After changing the prompt, score it on other code as well as those ten cases. The revised wording earns its place when it improves the review without concealing bugs the earlier version found.

Next: structured output and validation loops — guaranteeing schema-compliant output with tool use, and the retry loops that fix what schemas can’t.