Appendix A: Router Laboratory Reference
Dependency roles, inference calls, example-selection rules, and full reports for the prompting baseline in Chapters 4–5.
The first router experiment leaves two things to inspect: individual answers and a report about the run. Chapters 4–5 teach how to produce and interpret them. This appendix preserves the implementation details and complete reports for reproducing that experiment or investigating a particular setting.
For the first reading, follow Chapter 4’s saved-answer activity and Chapter 5’s correct-answer counts. Return here for library and inference details or example selection and full reports.
Run against: The recorded baseline used Python 3.13.3, PyTorch 2.8.0, Transformers 4.56.2, Datasets 4.1.1, PEFT 0.17.1, TRL 0.23.1, and scikit-learn 1.7.2 on Apple M1 Ultra/MPS. Qwen/Qwen3-0.6B revision c1899de289a04d12100db370d81485cdf75e47ca, dataset router-v1, policy bookshop-routing-v1, run baseline-v1.
Not run. This baseline does not include router training, CUDA or CPU-only Qwen benchmarking, concurrent serving, production traffic, or a true MPS accelerator-memory peak. Its synthetic teaching fixtures have no independent human annotation. Training experiments follow in Chapters 7–9.
Laboratory implementation
The companion project holds the files below. The environment setup and first model response are explained in Chapter 4; the excerpts here use that runner’s objects and configuration.
Project files and installed tools
fine-tuning-specialists/
├── requirements.txt
├── requirements-frozen.txt
├── model.json
├── build_data.py
├── router.py
├── mechanics.py
├── smoke.py
├── summarize.py
├── test_router.py
├── data/
│ ├── demonstrations.jsonl
│ ├── development.jsonl
│ └── test.jsonl
└── runs/
└── baseline-v1/
├── predictions.jsonl
└── results.jsonsmoke.py checks one model response, and summarize.py reads a saved summary. mechanics.py holds the single-weight calculation from Chapter 3 and an optional adapter demonstration. build_data.py reconstructs the original fixtures; test_router.py checks the baseline runner. The environment and downloaded model cache are reconstructed locally rather than committed with the project.
The smaller requirements.txt lists direct dependencies. requirements-frozen.txt also records their transitive dependencies, the packages those libraries need. Keeping both makes the intended tools and the complete recorded environment visible.
PyTorch performs the model’s numerical calculations. It stores values in tensors, arrays of numbers with a shape. Transformers loads the language model and tokenizer and provides text generation. The Hugging Face Hub hosts the checkpoint files identified by the model name and revision.
Scikit-learn handles the word-based TF-IDF example selector and the scoring calculations. The selector is a separate program that chooses examples — not another language model being trained alongside the router.
Other packages are installed for later work. Datasets supports loading and transforming training data. PEFT, short for Parameter-Efficient Fine-Tuning, supplies adapters such as LoRA. TRL supplies supervised fine-tuning tools. The setup checks import the training classes and construct an adapter, but the baseline itself does not train a model.
| Component | Pin | Role in the first five chapters |
|---|---|---|
| Python | 3.13.3 | Interpreter |
| PyTorch | 2.8.0 | Inference and mechanics demonstrations |
| Transformers | 4.56.2 | Tokenizer, checkpoint loading, generation |
| Datasets | 4.1.1 | Installed for the training data path |
| PEFT | 0.17.1 | Adapter construction and parameter inspection |
| TRL | 0.23.1 | Training-class import check |
| scikit-learn | 1.7.2 | Example selection and metrics |
These are the versions the experiment used, not the newest releases; if you deliberately choose newer packages, preserve the resulting environment and treat it as another configuration to check.
The smoke test constructs a rank-8 LoRA adapter and counts 1,146,880 trainable values alongside the base model’s 596,049,920 parameters. It also imports the training classes and prints package versions. These checks prepare for later chapters; they do not run training or establish adapter quality.
Numerical and attention settings
float32 stores each parameter value in a 32-bit floating-point format when loading it for this experiment. The representation of downloaded files and the representation used in memory need not be the same, and the explicit setting tells us which one this run requested.
The eager setting chooses the implementation used for the model’s attention computation, which is part of how the model uses its input positions to calculate the next token. You do not need its internal mathematics to run this comparison; keep the implementation fixed so it does not become another changing variable.
The message-to-token call
This excerpt uses the companion’s already-loaded tokenizer, model, selected device, and messages_for helper. It is part of the runner, not a standalone script to paste into an empty Python session:
inputs = tokenizer.apply_chat_template(
messages_for(request_text, examples),
tokenize=True,
add_generation_prompt=True,
enable_thinking=False,
return_dict=True,
return_tensors="pt",
).to(device)messages_for(request_text, examples) builds the conversation. tokenize=True asks the template call to return tokenized input rather than just formatted text. return_tensors="pt" puts that input into PyTorch tensors, and .to(device) moves them to the selected device.
add_generation_prompt=True leaves the sequence ready for an assistant continuation, and enable_thinking=False is the checkpoint-specific choice already recorded in the configuration. The resulting input length includes the policy, examples, request, and conversation markers — counting only the visible request would miss much of the model’s input.
The Transformers chat-template guide explains this message-to-token boundary. Keeping the exact call in the companion avoids asking each reader to reconstruct that boundary independently.
Generation controls and extracting the answer
The next operation produces output tokens. Checkpoint files can carry generation defaults, so the runner passes its choices at the call site. In particular, it disables sampling and the settings used only for sampling:
model.eval()
model.requires_grad_(False)
with torch.inference_mode():
output = model.generate(
**inputs,
do_sample=False,
temperature=None,
top_p=None,
top_k=None,
max_new_tokens=12,
eos_token_id=tokenizer.eos_token_id,
pad_token_id=tokenizer.pad_token_id,
)The setup lines have distinct jobs: eval() selects evaluation behavior in model components that distinguish training from evaluation, freezing parameters says that they are not being trained, and inference_mode() avoids constructing the ordinary graph of operations used for gradient calculations. None of these replaces the others.
The temperature, top_p, and top_k options control sampling behavior when sampling is used; here they are set to None because this comparison uses greedy selection. The end-of-sequence token marks a possible end to generation, and the padding token supplies the checkpoint’s padding identity. These settings remain fixed across prompt methods.
The Transformers generation reference distinguishes new-token limits from total sequence limits. Our twelve-token budget bounds the continuation, but the parser still has to check what the model actually returned.
The returned sequence contains the prompt followed by the continuation. We must remove the prompt portion before parsing, because the policy and demonstrations already contain valid labels:
continuation = output[0, inputs["input_ids"].shape[-1]:]
raw = tokenizer.decode(continuation, skip_special_tokens=True)
prediction = parse_label(raw)The slice starts at the input sequence’s length. decode converts only those new token IDs back into text, omitting special tokens, and parse_label checks the result. The original decoded string is saved even if it fails the label check, so we can distinguish an invalid response from an allowed label sent to the wrong destination.
Timing a completed request
The runner first warms all four prompting methods with a development request, then measures test requests while keeping the model loaded. This separates the repeated request path from model-loading work.
The measured interval includes example selection, chat formatting, tokenization, generation, and conversion back to text. Accelerators can schedule work asynchronously, so the timer waits for device work at the interval boundaries — otherwise it might measure the scheduling of a calculation rather than its completion.
Within each repetition, the request/method jobs are shuffled using a recorded seed, a value controlling the pseudorandom ordering. Mixing methods avoids always timing zero-shot first and twenty-shot last; it reduces that ordering bias without eliminating other desktop activity or caching effects.
Reading the memory counters
The record separates the space occupied by the weights from the memory used by the running process:
| Measurement | Recorded value | What it covers |
|---|---|---|
| Float32 parameter tensors | 2,384,199,680 bytes | The parameter count times four bytes per value; weights only |
| Process peak resident set (peak RSS) | 5,073,092,608 bytes | The most resident memory the whole process held at any moment in its lifetime |
Accelerator allocation peak (peak_memory_bytes) | null | Unavailable: the harness has no true MPS allocator-peak counter |
Peak RSS is broader than the weights, but it is not the accelerator’s own allocation peak. Substituting either measured value for the missing counter would answer a different question. On a unified-memory machine the counters also describe overlapping memory, so adding them would count some storage twice.
Baseline reproduction and reports
The following details accompany the quality and resource comparison in Chapter 5. They refer to the same saved experiment, not an additional run.
The answer key and example selection
The expected label in a test record is the answer we compare with the model’s response, so that label needs a clear rule behind it. It has one — the routing policy from Chapter 2. What the policy adds for a request that needs more than one source is a priority: a supplied document before private records, then external web evidence, with arithmetic following retrieval when its inputs are missing. That is this application’s rule for choosing a first destination.
Compare three exchange-rate questions. A rate printed on a receipt requires DOCUMENT; the current rate requires WEB; multiplication using a supplied rate requires CALCULATOR. Shared words do not settle the destination — which is why we need the policy before interpreting a score.
The test contains short, single-request cases. Fifty of them cannot establish that every mixed-source request, conversation referring to an earlier message, or missing attachment is handled correctly; those would require additional cases and application checks.
Chapter 4 checked the three collections: 40 demonstrations, 10 development cases, and 50 test cases, equal per class, with no request text repeated across them after normalizing case and whitespace. Held-out means, here, that the test requests stay out of the demonstrations and out of the text used to fit the example selector. The test is run only after the settings are fixed, and all four methods receive the same fifty questions and the same routing policy.
The fixtures are synthetic, with a provenance and a rationale for each expected label but no independent annotator, and duplicate checks cannot prove the collections share no phrasing or authoring assumptions. A small, reproducible comparison can still reveal a route the model never finds, a broken parser, or an unexpectedly weak prompting method — as long as the conclusion stays tied to the cases that were actually tested.
An experimental arm is one version of the method being compared. Our four arms change which demonstrations appear between the shared policy and the new request:
| Method | Demonstrations | Selection rule |
|---|---|---|
| Zero-shot | 0 | Policy and request only |
| Five-shot | 5 | First five interleaved records, one per class |
| Twenty-shot | 20 | First twenty records, four per class |
| Dynamic five-shot | 5 | Highest TF-IDF similarity from the forty-record pool |
“Interleaved” means the stored examples alternate through the destinations: the first five include one of each, and the first twenty include four of each. The dynamic method uses the TF-IDF wording comparison introduced in Chapter 2. It compares words and adjacent word pairs, giving rare shared terms more influence than terms common throughout the collection. Its vocabulary is fitted only on demonstration text; equal scores are resolved by stable example-ID order. If no words overlap, every score is zero and the ID order still returns five examples. The selector never receives the expected test label, and each prediction records the selected example IDs so we can inspect the prompt that led to it. The TF-IDF vectorizer reference describes the library’s text and normalization controls; the companion preserves the actual settings.
All arms use the common settings from Chapter 4: the pinned tokenizer’s chat template, thinking disabled, greedy generation, a twelve-token answer budget, and the prompt removed from the returned sequence before parsing. Holding these settings fixed lets us compare the demonstration strategies without also changing the generation configuration. A different policy, decoder, or output constraint may produce another result, but it would need its own comparison.
Reproducing the saved experiment
The companion already contains the baseline-v1 results discussed here, so you can inspect them without running the model again. To make your own comparison, use a fresh output directory from the companion root:
.venv/bin/python router.py --validate-only
.venv/bin/python router.py --device mps --split test --repeats 3 --output runs/local-baseline
.venv/bin/python summarize.py runs/local-baselineIf runs/local-baseline already exists from Chapter 4, select a different output name in both commands; the runner protects existing directories so reproducing the experiment does not overwrite the previous evidence.
There are 600 prediction records: fifty requests, four methods, and three repetitions. For each method, quality scores use the first repetition’s fifty unique requests, and timing summaries use all three repetitions — 150 observed request durations per method. Asking the same question again supplies another timing observation, not another unique question. Every repeated case received the same label in this run, which establishes stability under this rerun procedure, not cross-device determinism.
The warm-up, shuffled request order, and timer boundaries are described under Timing a completed request above. The runner saves raw outputs as it goes, so an interrupted run may leave a partial prediction file; a completed results.json is written only after all predictions have been collected and scored.
Full quality report and scoring conventions
The saved report also includes macro precision and macro recall. Each averages the corresponding class-level measure with equal weight per class, just as macro-F1 averages the class F1 scores. Macro-F1 is not calculated by inserting macro precision and macro recall into the F1 formula.
| Method | Correct | Accuracy | Macro precision | Macro recall | Macro-F1 |
|---|---|---|---|---|---|
| Zero-shot | 23 / 50 | 46% | 0.336 | 0.460 | 0.354 |
| Five-shot | 34 / 50 | 68% | 0.791 | 0.680 | 0.674 |
| Twenty-shot | 38 / 50 | 76% | 0.810 | 0.760 | 0.756 |
| Dynamic five-shot | 32 / 50 | 64% | 0.670 | 0.640 | 0.645 |
Macro recall equals accuracy here because every class has the same number of test cases. With unequal class counts, accuracy gives larger classes more weight, so the two measures need not match. Macro precision and macro-F1 tell different stories because each destination receives a different mix of right and wrong requests. Fixed five-shot, for example, has relatively high macro precision but lower macro recall: a destination predicted only a few times can be fairly precise while missing much of its intended work. We need the class-level counts to understand that tradeoff.
The scorer uses zero when precision or recall would otherwise be undefined because a class has no predictions or no test support, and it keeps the INVALID column. If a future model generates an explanation instead of one label, that request remains in the accuracy denominator and counts against recall for its intended destination.
The slower requests and the overall serial rate
The median describes the middle of the timing sample. p95, the 95th percentile, describes its slower end. Serial requests/s divides the number of requests by the sum of their measured durations. That rate uses all durations, so it is not the reciprocal of the median latency.
| Method | Mean input tokens | Median ms | p95 ms | Serial requests/s |
|---|---|---|---|---|
| Zero-shot | 241.62 | 195.2 | 260.6 | 5.60 |
| Five-shot | 351.62 | 217.3 | 273.3 | 4.87 |
| Twenty-shot | 692.62 | 287.6 | 376.8 | 3.57 |
| Dynamic five-shot | 357.80 | 170.0 | 290.1 | 5.07 |
The dynamic method has a lower median than zero-shot in this run, despite its longer input and selection work, while its serial request rate is lower and its p95 is higher. There is no contradiction: the middle observation does not determine the average or the slow end of the distribution.
The serial rate excludes logging overhead outside the measured intervals, and it is not an estimate of a server processing batches or simultaneous requests; that is another workload. The complete measured benchmark loop took about 129.5 seconds, and loading took about 1.12 seconds with the model files already cached, not freshly downloaded. Starting on a newly provisioned machine, calling a remote API, and serving concurrent traffic are different timing questions.
Comments