Prompt, Retrieve, Fine-Tune, or Scale?

A bookshop assistant gives the wrong stock count, prompting suggestions for a larger model or a fine-tune on inventory questions. Neither supplies the missing current inventory — that requires a fresh, authorized source of facts. Now suppose the assistant reads the right invoice but copies its buyer into the vendor field: the evidence is present, while the interpretation of roles is wrong.

These failures could produce equally disappointing accuracy scores, yet they need different repairs. Better examples, a clearer prompt, or training may address field selection; access to current records addresses the inventory question. Choosing a technique begins with locating what the system lacks — and identifying the part that can supply it.

Run against: The frozen baseline-v1, training-v2, and specialists-v1 results in the companion; Qwen3 models with the pinned environments and measurement boundaries recorded in their respective chapters.

Not run. A production RAG service, exhaustive prompt search, every combination of retrieval and training, and a common independent benchmark across all three tasks are outside these experiments.

The comparisons below preserve each study’s dataset and timing boundary. Scores from different task versions are not combined into one leaderboard.

Classify the missing ingredient

A failure can come from missing instructions, examples, evidence, learned behavior, or capability under the available budget. The categories can overlap, but naming the dominant cause gives the next experiment a purpose and a comparison to beat.

Missing ingredient	What it looks like	Intervention	What shows it worked
Instructions	The prompt never explains ambiguous dates, so choices are inconsistent	A clearer task contract	Fewer disagreements between evaluator and application on the same cases
Examples	The schema, label boundary or output style is hard to state abstractly	Demonstrations, fixed or selected per request	The boundary cases the examples show stop failing
Evidence	The answer needs a current fact the model was never given	Retrieval, database access or another tool under the caller’s permissions	The source path delivers the fact and the answer uses it
Learned behavior	A stable behavior repeats and carrying its examples in every prompt is costly or unreliable	Fine-tuning	The adapter matches the prompted baseline on the same cases with the shorter prompt
Capability	Task and data are sound, yet the model cannot learn the distinction under the available budget	Comparing stronger bases	A larger base repairs the named failure under the same recipe

Training on one interpretation of an ambiguous rule without updating the contract can conceal the ambiguity, leaving the evaluator and application to enforce different rules. These diagnoses guide experiments — they are not guarantees that the first chosen intervention will work.

Use a prompt to repair an unclear contract

The book’s router policy distinguishes the required source from keywords: explaining a database belongs to GENERAL_LLM, while reading private sales records belongs to DATABASE. Without that distinction, examples containing “database” can teach a lexical shortcut instead of the application’s intended routing rule.

A clearer policy gives both model and evaluator a shared contract, making disagreements easier to diagnose. Prompt changes still need versioning and regression checks, because improving one boundary can damage another — the expected labels must continue to express the intended task.

Repeated “except when…” clauses can signal that the policy needs a clearer hierarchy or narrower scope. They also consume context and may conflict with examples, so more instruction text is not automatically a better specification. When a compact prompt already meets the quality and cost requirements, that provides a working baseline against which training must justify its additional lifecycle.

Use demonstrations to show a boundary

Examples communicate details that are awkward to specify abstractly: an invoice can show decimal strings and JSON null, while a router example can show why arithmetic after retrieval still begins at DOCUMENT. Choose demonstrations for these distinctions, because five near-identical stock lookups do little to teach source precedence.

Fixed demonstrations make the prompt easy to audit, but every request carries them even when their lessons are irrelevant. Dynamic selection trades some simplicity for context matched to the request — both approaches still depend on whether the chosen examples teach the missing decision.

On the clean extractor test, fixed and dynamic five-shot prompting each recover all twenty-four records, establishing a strong baseline for an adapter. On the messy challenge, each obtains only thirteen correct records out of thirty-two. The useful next examples therefore concern ambiguity and conflict — repeating successful clean layouts does not address the observed gap.

Know which kind of retrieval you mean

Dynamic few-shot prompting retrieves examples of how to perform a task; retrieval-augmented generation retrieves evidence needed to answer the current request. Both add context — but recording them as one undifferentiated “RAG” method hides which missing ingredient they supply.

The companion’s TF-IDF selector searches labeled training requests for similar wording and returns demonstrations. An evidence retriever would instead fetch the current stock record, supplied invoice, or relevant handbook section, requiring checks of relevance, completeness, freshness, and access. A model cannot use an omitted source, and a correct answer from an unauthorized document still violates the application’s contract.

A system may need both forms of retrieval: the current invoice supplies facts, while extraction examples show how to represent them. Keeping these roles distinct in the prompt helps prevent an example’s amount or vendor from being mistaken for evidence about the current document. Evaluate the source path separately so a retrieval failure is not automatically diagnosed as weak generation.

Use training for behavior that repeats

Fine-tuning changes parameters so a repeated behavior can be reproduced without carrying all its examples in every request. Shorter prompts can make a stable task easier to serve, in exchange for maintaining a dataset, trained artifact, evaluation, and release process.

The router’s full fine-tune, LoRA, and QLoRA arms all reach the fifty-case router-v2 test’s ceiling. Their resource profiles differ — that test cannot rank their quality further. The extractor gives a more useful behavioral contrast: clean LoRA reaches perfect clean accuracy while obtaining only fourteen correct challenge records out of thirty-two.

Planned messy augmentation reaches all thirty-two under the shared family construction, showing the benefit of teaching uncertainty handling in this fixture. It also adds optimizer steps under the fixed-epoch recipe, so the gain cannot be assigned solely to the examples’ diversity. Chapter 12’s matched-step repair illustrates how to investigate that distinction separately.

The choice of training data determines which behavior receives practice, including when to abstain. Lower loss on tidy invoices cannot replace missing examples of ambiguity, just as accurate stock-question routing cannot supply the current stock count.

Keep the book’s comparisons in their own lanes

The original router baseline compares prompting strategies on router-v1, while training uses router-v2 with newly held-out families. Extraction and SQL introduce their own contracts and metrics, so each result belongs to the question its dataset was designed to answer.

Study	Comparable arms	What the result supports
Router baseline	Zero, five, twenty, dynamic five	Prompting choices on the same 50 cases
Router training	Base, full, LoRA, QLoRA	Adaptation under the router-v2 recipe
Router size sweep	Six data sizes plus repetition control	Fixed-epoch behavior on a narrow test
Clean extractor	Zero, fixed five, dynamic five, clean LoRA	Known layouts with new values
Messy extractor	Prompting, clean LoRA, augmented LoRA	Eight declared perturbation families
SQL scaling	Four quantized bases and adapters	Same query families and training budget

A training gain requires base and adapted predictions on the same cases. Subtracting a router-v1 baseline score from a router-v2 trained score changes both the model and evaluation population; use the router-v2 base arm for that comparison.

Cross-task percentages also have different meanings: a six-field record has more ways to fail than one route label, and SQL must match results on two fixtures. Timing adds another boundary, because the router baseline includes demonstration selection while the extractor starts timing after selection. Preserve these distinctions when using several experiments to support a decision.

Scale only after the smaller interventions have a fair test

The SQL sweep’s smallest adapter answers twenty-two of twenty-four requests correctly, while the 1.7B adapter answers all twenty-four. Chapter 16 locates the difference in two anti-join requests and connects their repair to the additional latency, giving a concrete capability to weigh against cost.

If those requests matter to the product and remain unreliable after narrower repairs, the larger model deserves a stronger evaluation. If the supported scope excludes that operation, a restricted smaller specialist may be sufficient. Each size’s base evaluation also matters — a stronger base may need only prompting, while another needs task-specific training to reach the same requirement.

The fixed recipe compares one allocation of learning rate, adapter scope, and training budget across sizes. Investigate alternatives when the cost difference is material or a candidate sits near a requirement boundary, rather than treating this ranking as permanent.

Different paths can use different capacities: a small router can dispatch difficult SQL to a stronger model while routine extraction stays local. The application needs reliable components for its supported work, without requiring every component to have the same size or training method.

Walk through three bookshop decisions

“How many copies of this title are in stock right now?” requires current private records. A prompt may clarify the request and a router may choose DATABASE, but an authorized source query supplies the missing count. After that source path works, evaluate whether the answer uses its returned evidence correctly.

“Extract the vendor and total from this supplied invoice” already includes its evidence. Start with a clear schema and prompt, then demonstrations that distinguish the roles; compare an adapter when recurring volume makes those examples expensive or their behavior remains unreliable. The useful comparison keeps the documents and acceptance policy fixed.

“Rank every book within its category by price, preserving all rows” requires a window calculation over the supplied schema. If a small model instead collapses rows despite appropriate examples, comparing a stronger base tests a capability hypothesis. This is a hypothetical failure to investigate — the recorded window-family requests pass for every adapted size.

These requests locate different missing ingredients, yet a single path can combine the remedies. It might retrieve an authorized schema, select a demonstration, invoke a trained SQL specialist, and send a rejected result for review. Each additional stage should address a measured failure because it also adds latency and another place where the request can go wrong.

Write a decision that can be reversed

A decision record ties the proposed intervention to a task, observed failure, comparison, and rule for revisiting the choice. The following worked record uses the extractor’s measured results, with an explicitly illustrative release requirement. It proposes a next evaluation rather than treating a synthetic score as permission to import real invoices.

Decision field	Worked invoice decision
Supported task	Extract six invoice fields; use null when the source is missing or ambiguous
Current baseline	Fixed five-shot prompting: 24/24 clean and 13/32 challenge exact records
Failure to address	Ambiguity and conflicting facts remain unreliable under the baseline
Candidate	Planned augmented LoRA: 32/32 exact on the shared-family challenge
Comparison	Same documents and strict evaluator for prompting, clean LoRA, and augmented LoRA
Acceptance gate	Illustrative: preserve 24/24 clean records and reach 32/32 challenge records; require a separate reviewed real-document reserve before release
Remaining evidence	Check the candidate’s clean predictions and evaluate the new reserve; isolate extra-step effects with a matched-step control if attributing the gain
Revisit or reverse	Return to prompting plus review if the schema changes, accepted-output errors rise, or review cost removes the expected savings

The challenge result nominates a candidate — the gate states what still needs checking before release. In particular, the perfect clean score quoted for the baseline must not silently stand in for the augmented adapter’s clean behavior. This record makes that next check visible instead of borrowing a neighboring arm’s result.

Now complete a second record for a tool that must answer every declared SQL family correctly and keep median inference below five seconds. Use Chapter 16’s measured counts and medians, name the eligible candidate, and identify the evidence needed beyond those twenty-four requests. Exercise 4 gives a worked answer, then asks how a fixed report catalogue might change the implementation.

A reversal condition keeps this reasoning useful after the first deployment. Keeping the baseline implementation and its evaluation contract available lets the team repeat the comparison when the schema, workload, or operating costs change.

Compare operational change cost as well as inference cost

A prompt revision is lightweight to package, whereas an adapter revision adds training and artifact handling. Retrieval changes may require indexing and access-policy checks, while a larger model can change hosting capacity and startup behavior. The frequency of these changes matters alongside the cost of one inference request.

For a frequently changing schema, updating a prompt and validator may be easier than recurring retraining. For a stable schema at high volume, an adapter’s shorter prompt may justify that maintenance. Reliable labels remain a prerequisite: a prompt prototype can expose disagreements before they become thousands of inconsistent training rows.

Each technique also leaves a different debugging record. Demonstrations are visible in a request trace — learned behavior requires dataset provenance and controlled comparisons to investigate. Version coupled components together so rolling back an adapter after a schema change does not leave its validator expecting an incompatible record.

Chapter 19 turns these choices into cost arithmetic, including review work and fallback alongside inference. A lower model bill becomes useful only when the resulting workflow still meets its quality gate — a decision record preserves that condition when prices or serving tools change.

Know when a deterministic solution wins

Some of the book’s tasks are narrow enough for conventional software to provide a serious baseline. Fixed invoice layouts can be parsed, supplied operands can go to a calculator, and known SQL reports can use parameterized queries. A language model becomes useful when variation in wording or structure makes those direct mappings expensive to maintain.

One hybrid uses the model to identify intent and extract parameters, then validates them before executing an approved operation. This reduces the amount of unconstrained generation that can affect a result, while preserving the need to evaluate semantic interpretation. An approved query for the wrong category still answers the wrong question.

Compare that hybrid with free-form generation using the actual requests and task requirements. The aim is to assign each operation to a component that can perform it reliably, leaving the model the language variation that ordinary rules struggle to handle.

Exercises

1. Current stock

A model gives an outdated inventory count. Which intervention should be investigated first?

Answer

Check the source path: authorized current inventory data must reach the answer process. A larger model or fine-tune cannot reliably substitute for changing private facts. After retrieval works, evaluate whether the model uses the returned evidence correctly.

2. Example retrieval versus evidence retrieval

A TF-IDF selector returns five labeled invoice examples. Has the system retrieved the facts of the current invoice?

Answer

No. It retrieved demonstrations of the task. The current invoice is a separate evidence source. Keep those roles distinct in prompts, evaluation, and reporting so an example’s values cannot be mistaken for the current document’s facts.

3. Cross-version arithmetic

Why is subtracting the original router baseline score from the later router-v2 trained score invalid as a controlled training gain?

Answer

The datasets differ. Use the base and trained arms evaluated on the same router-v2 cases for that comparison. Different studies can inform a decision without being combined into a misleading before/after number.

4. Complete the SQL decision record

An illustrative reporting tool requires 24/24 execution-correct test answers, 24/24 valid queries, and median model inference below five seconds. Complete the candidate, comparison, remaining-evidence, and reversal fields using Chapter 16. Would a product limited to six fixed report types necessarily need generated SQL?

Answer

The 1.7B adapter is the smallest measured candidate meeting these requirements: 24/24 correct and valid, with a median near 4.35 seconds. The 0.6B adapter is faster but misses two answers; 4B and 8B exceed the median limit. Compare them on the same requests and timing boundary, then evaluate fresh supported requests and the complete application path before release. Revisit the choice if that fresh quality or latency fails the gate, or the supported schema changes.

Six fixed reports create another candidate: classify intent, extract validated parameters, and execute approved queries. Keep that implementation available as a comparison or reversal path if its measured task quality and cost are better. Meeting a model-size gate does not require using generative SQL when a simpler implementation meets the product contract.

Final thoughts

Our experiments give us several ways to repair a missing ingredient, along with reasons to check that the repair worked. They show ceilings, narrow tests, and cases where prompting is already strong. Carry those limits into your decision; a successful run supports a particular choice, not a universal rule.

Next: Building the Multi-Model Specialist System