Building Training Data

“Calculate our return rate from the shop’s transaction database.” A router that sees calculate and chooses CALCULATOR has learned the wrong boundary. The inputs it would need live in private records, so the first destination is DATABASE.

A useful training collection puts that boundary beside a contrasting case: “Given 100 sales and 5 returns, calculate the return rate.” The second request supplies the operands, so it belongs to CALCULATOR. Together the two examples teach the role of evidence rather than a shortcut based on a verb.

Start with these distinctions when designing the data — and count the rows afterward. A thousand variations of “add two numbers” will do little for a request that needs those numbers retrieved first. A smaller collection that exposes the decisions the router actually has to make gives more useful supervision than a larger one that repeats the easy ones.

Chapter 5 identified the mistakes we want to address. This chapter turns a routing decision into a training row, keeps related examples together when dividing the data, and follows one row into the two lists the training code needs. You can work through the examples with a text editor, and the one activity that runs code needs only Python’s standard library.

The bookshop router still predicts one of five exact labels under the policy from Chapter 2. A supplied document comes before private records, and private records before the web, when several sources are required; arithmetic follows retrieval; general explanations, creative requests and requests needing clarification go to GENERAL_LLM. Training changes the model’s behavior under this contract; it does not quietly replace the contract.

Run against: Python 3.13.3, Transformers 4.56.2, and the pinned Qwen3-0.6B tokenizer; companion dataset router-v2, training_data.py, curated.py, and the completion encoder in train_router.py. The row-counting activity ran with Python 3.13.5 and the standard library against the companion’s committed data/router-v2 files.

Not run. Independent human annotation, representative customer sampling, multilingual coverage, and semantic leakage certification are outside this teaching fixture.

Start with a decision that two people can explain

An example needs more than a plausible label; it needs a reason someone else can inspect. Consider “What did the customer pay?” That text alone leaves the source unspecified. One designer might imagine an attached receipt while another imagines a customer database, and both can defend their answer, because they are labeling different imagined situations.

Adding “according to the attached receipt” makes the source explicit — and adding “in our stored transactions” makes a different route explicit. When neither is present, the policy should say whether the router asks for clarification or uses conversation context. Our short, single-request fixture has no hidden attachment state or conversation history, so a request cannot earn a label from information that is absent from its text.

When reviewers disagree, the first question is whether someone chose the wrong label or whether the request leaves the decision open. Record which source the text requires and why. If resolving the disagreement needs a new rule, add it to the policy before labeling more examples — otherwise the next hundred rows inherit the ambiguity.

The companion retains the original bookshop-routing-v1 policy. As in Chapter 1, choosing a destination leaves access checks and validation of downstream results to the application.

Give each row an inspectable shape

A training row pairs a request with the destination we want the model to produce, and it carries a few more fields that help us inspect and organize the collection. The model receives the request and policy as input; the label supplies the desired answer, called the completion.

For the first curated database example, the essential content is:

{
  "id": "curated-DATABASE-00",
  "split": "train",
  "label": "DATABASE",
  "text": "Which of our orders are still waiting for payment?",
  "family": "train-curated-DATABASE",
  "rationale": "Private operational records supply missing facts.",
  "provenance": "Individually AI-authored teaching request; no independent annotation"
}

Read text and label first: the shop’s stored orders supply the missing facts, so the answer is DATABASE. The rationale explains that decision to a reviewer — it stays outside the model’s answer, because our parser still expects just the destination.

The remaining fields keep the row traceable. id names it, so a wrong prediction can be traced even if the wording is later revised. A revision should create a new dataset version rather than silently change a frozen row. split assigns the row to a collection such as training, and provenance records where the example came from. family groups related examples, which matters when we separate training from testing below. The word curated here means individually composed and inspected for the policy; it does not mean independently human-annotated ground truth.

Where the 150 rows came from

The core has 150 individually authored teaching requests, thirty for each label, covering inventory lookups, invoice retrieval, document extraction, current external facts, supplied arithmetic, explanations, creative writing and clarification. They were written one at a time with an AI assistant, prompted for a destination and a boundary rather than for volume. Each one was read against the policy before it was kept, with the boundary table in the next section as the checklist. That is a process a practitioner can copy for a new task in an afternoon. Decide the boundaries, write a few dozen requests per class that sit on each side of them, and reject every request whose label needs a fact the text does not contain.

The rows are short and deliberately legible — and they are not a sample of a real support queue. Chapter 11 takes the same idea to scale, where a model generates candidates and filters do the first pass of the reading.

Cover boundaries before adding paraphrases

For each decision, put examples side by side that would expose a failure to distinguish the required sources. This table makes the intended coverage visible:

Boundary	One side	Contrasting side
Missing arithmetic inputs	Compute profit from our ledger: DATABASE	Revenue 120, costs 80: CALCULATOR
Source versus subject	Explain databases: GENERAL_LLM	Look up our customer record: DATABASE
Document versus live source	Rate printed on uploaded receipt: DOCUMENT	Current exchange rate: WEB
Creative versus current facts	Invent a review: GENERAL_LLM	Find recent published reviews: WEB
Quoted instructions	Summarize a memo saying “search online”: DOCUMENT	Explicitly search online for a source: WEB

Easy examples establish the centers of the classes and boundary examples establish their edges — a collection needs both. One that contains only hard cases omits the ordinary wording that dominates real use. One that contains only easy cases can produce an impressive loss curve while preserving exactly the shortcuts that motivated training.

Negative examples belong beside the shortcut they challenge. “Explain what a relational database is” is a negative example for the rule “database means DATABASE.” It is also a positive supervised example for GENERAL_LLM — there is no separate negative-label syntax in a five-class task.

Ambiguity also needs a deliberate home. “Help me with my book” identifies neither a retrieval source nor a calculation, so under this policy the general model can ask what help is needed. A production dispatcher might instead expose an explicit CLARIFY destination. That changes the label vocabulary, parser, demonstrations, training data and evaluation together; adding the label in a few training rows alone would create incompatible interfaces.

Mixed requests are the easiest to label inconsistently. “Read this attached invoice and check whether it matches our records” requires more than one operation, and our single-label policy chooses the supplied document first. That makes the first step deterministic without proving that a one-step router can finish the workflow, which a later orchestration layer would have to carry.

Write and check a contrasting pair

In a scratch file, write two requests about refunds: make one require the shop’s records and give the other all the numbers needed for a calculation. For each, add a label and a one-sentence rationale, and keep these practice rows separate from the companion’s frozen data.

Before reading the example below, hide your labels and choose the routes again from the request text alone. Can you point to the words that establish where the answer comes from? If your reasoning needs a fact you did not write, revise the request.

One possible pair is:

Request	Label	Rationale
What was the total refunded according to our stored transactions yesterday?	DATABASE	The amounts must be retrieved from private records.
Yesterday’s refunds were 12 and 8 pounds. Add them.	CALCULATOR	The request supplies both amounts.

Both requests concern refunds, but they teach different first steps. Now change only yesterday to Monday in the first request. Its route stays the same, because you have added wording variation; the second request added a different decision. Recognizing that distinction is what the next section is about.

Separate development from judgment

Chapter 5’s test failures are now visible and discussed in prose. Reusing those cases to select new examples would turn the old test into development material. It remains useful as a regression collection, but it can no longer honestly be called an untouched final test for the next training experiment.

The new fixture therefore creates three collections, each in its own file. Training examples guide weight updates. Validation examples help us check the training setup and make development choices, and test examples are reserved for judging the candidate once those choices are fixed.

A template is a sentence with a replaceable part, or slot, and filling the slot produces a complete request. For this invented illustration, take the template “Has order {order_id} been dispatched?”:

Slot value	Resulting request	Label
BK-907	Has order BK-907 been dispatched?	DATABASE
BK-214	Has order BK-214 been dispatched?	DATABASE

These two rows belong to one template family: they share a wording pattern and differ only in the order ID. If the first enters training and the second enters testing, there is no exact duplicate. Yet the test then largely asks whether the model recognizes the same sentence with a different identifier, which tells us little about unfamiliar wording.

So the template is assigned to a collection before its slots are filled. If it belongs to training, both rows above belong to training along with every other variant generated from it, and different templates are reserved for validation and test. The companion follows this order of operations, which is what prevents a training template from also supplying test variants.

Figure 06.1. Read upward. Splitting generated rows can put the same wording pattern on both sides of a training/test boundary. Assigning the family first keeps its variants together; validation and test use different families. Order IDs stand for the two complete requests above, both labeled DATABASE. This prevents shared template wording, not all semantic similarity.

Collection	Rows	Construction	Role
Authored core, the first 150 rows	150	Individually authored, balanced	Full/LoRA/QLoRA comparison
Expanded training pool	2,000 total	Core plus 1,850 template variants	Data-size experiment
Validation	50	Ten reserved template families	Development and warmup
Test	50	Ten different reserved families	Fixed final comparison

The training pool draws on 45 families: the five curated groups, one per label, and forty templates. The fifty test rows, by contrast, come from only ten families with five slot variations each. That is much less diversity than fifty independently sourced customer requests, since a model can succeed on every variant after learning one family. Separating families keeps template wording out of the wrong file; it does not make the three collections independent of the one author who wrote the policy and all of them.

Run the builder and read what it made

From the companion repository, the reproducible build is one command, and it prints a manifest as it finishes:

.venv/bin/python training_data.py

The manifest records, for each collection, its row count, family count, per-label counts and a SHA-256 hash of the file. This is its test block:

{
  "test": {
    "rows": 50,
    "families": 10,
    "labels": {
      "DATABASE": 10,
      "DOCUMENT": 10,
      "WEB": 10,
      "CALCULATOR": 10,
      "GENERAL_LLM": 10
    },
    "sha256": "ffcdc7402fafa1a9e78763ba8e786c821345108d59c12af50e7043f4fcbdccd5"
  }
}

The hash identifies the exact bytes a run used. A mismatch does not say which label is wrong — it says we are no longer discussing the same dataset, which is the question to settle before any other.

You can check the manifest’s claims yourself without the companion’s environment, because the files are JSON Lines like the ones Chapter 4 read. Save this as count_rows.py in the companion directory and run it with python3 count_rows.py:

import json
from collections import Counter
from pathlib import Path

for split in ("train", "validation", "test"):
    path = Path(f"data/router-v2/{split}.jsonl")
    rows = [json.loads(line) for line in path.read_text().splitlines()]
    labels = Counter(row["label"] for row in rows)
    families = {row["family"] for row in rows}
    print(f"{split:10} rows={len(rows):5} families={len(families):3} {dict(sorted(labels.items()))}")

train      rows= 2000 families= 45 {'CALCULATOR': 400, 'DATABASE': 400, 'DOCUMENT': 400, 'GENERAL_LLM': 400, 'WEB': 400}
validation rows=   50 families= 10 {'CALCULATOR': 10, 'DATABASE': 10, 'DOCUMENT': 10, 'GENERAL_LLM': 10, 'WEB': 10}
test       rows=   50 families= 10 {'CALCULATOR': 10, 'DATABASE': 10, 'DOCUMENT': 10, 'GENERAL_LLM': 10, 'WEB': 10}

The script prints the row, label and distinct-family counts for you to compare with the manifest. Matching counts do not identify the file: that requires computing its SHA-256 hash and comparing it with the recorded manifest, which this script does not do. Ten families in a fifty-row test gives an average of five rows per family; checking that every family has five requires counting rows within each family. Nor do separate family totals show whether a family appears in more than one file. The builder checks that membership across splits, as described next. Here, the printed counts let us spot a missing label or an unexpected collection size before training.

Validate what software can actually enforce

The builder rejects duplicate normalized text across the whole fixture, where normalization trims outer space, folds case and collapses runs of whitespace. It also verifies the label vocabulary, each row’s split, and that a recorded family belongs to only one split. These are useful mechanical assertions, because a missing class or an accidental duplicate can otherwise survive a successful training job. They are also the limit of what a schema can tell you. A row labeled DATABASE could still ask for a poem, and the file would remain valid JSON with a legal label.

Semantic review therefore asks different questions. Does the source named in the text support the answer? Are all the arithmetic operands supplied? Does a quoted instruction remain data? Would two routes be defensible under the current policy? Does the request depend on an attachment that the surrounding application never provides? Each of these needs policy reasoning, not a schema.

With a larger dataset, software that finds similar text can suggest requests to review together, but similarity is a prompt for reading rather than a verdict. Two nearly identical sentences can still need different labels when one says “attached receipt” and the other says “current website.” Deleting one for its wording would remove the contrast we are trying to teach.

Balance serves an experiment, not every deployment

Equal class counts make the teaching comparison easy to inspect. Each destination contributes the same number of examples, so a majority-class guess cannot win simply because one route dominates the file, and macro-F1 gives each class equal weight when the predictions are scored.

Real traffic may be heavily skewed — most requests might ask for order status while calculator requests are rare. A balanced test then answers how well the model handles a balanced task, not how often a random production request succeeds. Keep a balanced collection for comparing destinations, a separate representative collection for estimating everyday performance, and — if you need one — a third collection of requests chosen to test difficult boundaries. Each score then answers a clear question.

Repeating rare training examples changes how often the model learns from them, but it does not add new situations. Ten calculator requests repeated a hundred times still teach only those ten requests. Record how examples were selected and repeated alongside the row counts.

The companion takes its size-study collections for Chapter 10 as the first N rows of this file, each nested inside the next. Every consecutive group of five contains one request per label, so the 10-, 50- and 100-row collections hold two, ten and twenty examples per class. The first 150 rows are the full authored core, and the larger collections add template variants. Balance stays stable as the arms grow, while the remaining diversity limit stays explicit.

Preserve a dataset you can argue about

The companion stores the authored requests, the template builder, the materialized JSONL files and the manifest, and rebuilding the data is deterministic. The original baseline runner and router-v1 files remain unchanged, so a future comparison can identify which of the policy, the data, or the runtime changed.

Keep a short description with the files — this collection contains invented, short English requests for a teaching experiment, with a limited number of test families. Real customer interactions would also bring collection, privacy and retention rules of their own, and removing names alone can leave identifying details. This fixture does not demonstrate that process.

Review a batch before it becomes supervision

Review at two levels: whether each row’s text supports its label, and whether the batch as a whole covers the intended decisions. Those are different questions, because every row can be correctly labeled while the collection repeats one easy pattern and teaches little else.

Imagine receiving twenty new requests about invoices. Ten ask for values printed on supplied receipts, six ask for stored invoices in the shop’s system, and four supply amounts and ask for arithmetic. The subject is the same throughout, but the destinations are DOCUMENT, DATABASE and CALCULATOR. Labeling the batch by its topic would erase exactly the source distinction the router has to learn.

The row-level check is the one from the refund activity: hide the proposed labels, choose from the text, then compare. Revise any request that leaves out the information needed to choose. The batch-level check examines what is absent. Are any examples phrased as questions, and are some direct commands? Does the decisive source appear at the end of a request as well as at the beginning? Is there a counterexample to the most obvious keyword shortcut? Questions like these turn a vague wish for diversity into properties you can inspect.

The current fixture reuses one rationale string per class rather than recording a separate review decision for every sentence. In your own collection, keep the reason for a changed label and the policy version used to decide it. Set unresolved rows aside until the text or the policy is clear enough to support an answer.

Keep validation useful without turning it into a second training set

Validation lets us debug training and choose settings, and repeatedly adjusting a model to do well there makes its score optimistic — which is why a separate test exists. What matters is which examples influenced our choices — whatever a file happens to be named.

In the companion, validation supports small trial runs and warmup, and the plan chooses the final training checkpoint before the test is evaluated. Publishing those test results makes them available for later development, so any later repair based on them needs another fresh evaluation reserve.

Teach only the completion you want

Chapter 3 explained how training rewards the model for assigning more probability to the desired next token, and for our router those desired tokens spell the destination. Return to “Which of our orders are still waiting for payment?” We supply the policy and request, then teach the model to complete them with DATABASE and stop.

Separate the sequence into two parts before thinking about code:

Part	Content	Job during training
Prefix	Policy, request, and chat markers that begin the assistant’s answer	Supply the context used to predict the answer
Completion	Destination label followed by an end token	Supply the desired next tokens that we score

The end token marks where the answer should stop. We score it too, because a model that returns a label and then continues with an explanation has broken the interface.

The training code keeps two lists of the same length. input_ids holds the whole sequence as token IDs, and labels identifies the desired tokens used to calculate loss. Here labels means a list of token-level targets, while the row’s label field is the destination string; the two names are close, and the chapter keeps them apart on purpose.

For a small illustration, suppose the prefix has IDs [11, 12, 13], the destination has two tokens [41, 42], and the end token is 99. These IDs and lengths are invented to keep the lists readable; they are not Qwen’s tokenization of the request.

Position	1	2	3	4	5	6
Token’s role	Prefix	Prefix	Prefix	Label part 1	Label part 2	End
input_ids	11	12	13	41	42	99
labels	-100	-100	-100	41	42	99

In this loss function, -100 means “do not score a target here.” It is a marker in the target list, never an input token. This selection of scored positions is called a loss mask. The prefix stays available as context even though predicting its tokens contributes no loss.

Figure 06.2. Read the two rows position by position. Both lists have the same length, and only the last three entries of labels carry a target, so the loss is computed on the destination and its end token while the prefix remains visible to the model as context. The values are the chapter’s invented IDs, not a real tokenization.

The first scored prediction asks: after [11, 12, 13], how much probability did the model assign to 41? The next asks for 42 after the prefix and the desired token 41, and the last asks for 99 after both destination tokens. The model handles this next-token alignment internally, so we store the targets at their ordinary positions — as the table shows — rather than shifting the list ourselves.

Connect the lists to the encoder

Chapter 4’s chat template prepares the policy and request for an assistant answer. The encoder uses the same template and the same disabled-thinking setting, then appends the desired completion. This excerpt is what the training runner uses:

prefix = tokenizer.apply_chat_template(
    messages_for(row['text'], []),
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=False,
)
completion = tokenizer.encode(
    row['label'] + tokenizer.eos_token,
    add_special_tokens=False,
)
ids = prefix + completion
labels = [-100] * len(prefix) + completion

The last two lines build the lists we just inspected. prefix + completion joins the input tokens, and [-100] * len(prefix) creates one ignore marker for each prefix token, so adding completion to it preserves every desired answer token. The tokenizer call returns plain lists here, which is why + joins them rather than adding their values.

A destination such as GENERAL_LLM can occupy several tokens, and all of them need supervision, followed by the end token. The router still generates text — we have not replaced its output with a separate five-class classifier. Because the loss averages across scored tokens, a longer destination contributes more target positions than a shorter one.

That is the whole of what one row teaches. Training processes rows in batches, and rows of different lengths have to be padded to a common width before they share one array. That introduces a second mask with a different job, and Chapter 7 takes it up at the point where the batch is actually built.

Exercises

1. Repair an ambiguous row

A row says “What is the total on the invoice?” and has label CALCULATOR. What information is missing, and how would you create two unambiguous examples?

Answer

The source and operands are unspecified. “Read the total printed on the attached invoice” belongs to DOCUMENT. “The supplied invoice amounts are 20 and 3 dollars; add them” belongs to CALCULATOR. The original row should be clarified or removed rather than defended by an imagined context.

2. Find the leak

A generator creates fifty variants of each template. A random row split sends forty variants to training and ten to test. No exact text is duplicated. Why is that still a weak test?

Answer

The same wording skeleton appears in both collections. Changing a name or number often preserves the decision pattern. Splitting template families before rendering prevents this direct overlap. It still does not guarantee semantic independence or representative coverage.

3. Count the scored positions

A row has 240 prefix tokens, a three-token destination, and the end token. How many entries does labels hold, how many of them are -100, and what goes wrong if the encoder forgets to append the end token to the completion?

Answer

labels has 244 entries, one per input token: 240 markers of -100 for the prefix and four scored targets, the three destination tokens and the end token. Without the end token, only three positions are scored and the model is never taught where the answer stops, so a trained router can produce the right label and keep writing, which the strict parser rejects. The prefix still shapes every prediction as context; the mask only decides which positions are scored.

4. Interpret a balanced score

A router achieves 90% accuracy on a balanced five-class test. Production traffic is 90% DATABASE. Can the balanced score be reported as expected production accuracy?

Answer

No. The balanced score weights classes equally through their equal counts. Expected production accuracy depends on class frequencies and performance within each class, as well as differences inside the classes. A representative evaluation set is needed to measure that workload directly.

Final thoughts

I would spend the next hour resolving ten uncertain labels before generating another thousand easy requests. A model learns an inconsistency just as efficiently as it learns a useful boundary. The 150-row core gives us an inspectable starting point, and the reserved evaluation families give us a limited but explicit way to judge what training changes.

Next: Your First Full Fine-Tune