Production Fine-Tuning

A saved adapter becomes part of a production release alongside a particular base model, tokenizer, policy, validator, and data contract. If any of those pieces changes unnoticed, the adapter’s evaluation no longer describes the deployed system — it describes one that no longer exists.

We now have several candidate bookshop specialists and a way to measure them. To deploy one, we need to keep its evidence attached to the behavior we actually serve. An identifiable, reversible release lets us do that — without allowing a failed experiment to become the active service silently.

The lifecycle begins with traffic and ends with traffic: observe failures, curate examples, train a candidate, evaluate it, promote deliberately, and watch what changes. An observation may call for new training data, a prompt correction, a schema repair, or a permission fix. Identifying the cause before choosing an intervention keeps the release process focused on the behavior that actually needs to change.

Run against: Python 3.13.3; companion operations.py and test_specialists.py, including content-hash checks, quality and latency gates, deterministic canary assignment, atomic local pointer replacement, and rollback.

Not run. A live production deployment, distributed rollout, service load testing, customer monitoring, signed artifact distribution, and incident response with real traffic are outside this book’s local rehearsal.

The release helper demonstrates a filesystem-level control path. It is not a model registry, deployment platform, or complete supply-chain security system.

Version the behavior, not just the weight file

An adapter name such as invoice-v3 leaves out other pieces that determine its behavior when a request arrives. Changing the base revision can alter predictions, while changing the tokenizer or chat template can alter the sequence presented to the model. A validator update can turn yesterday’s accepted invoice into today’s rejection — even when all the model files stay identical.

The release identity should bind these behavior-defining inputs to the evidence used for acceptance, including evaluation code and dataset versions. Training settings explain how the candidate was produced, while serving settings describe the decoder and preprocessing used to answer requests. Recording both lets us check that the evaluated behavior matches the configuration we intend to deploy.

Release component	Why it matters
Base and tokenizer revisions	Define the pretrained model and token mapping
Adapter content hashes	Identify the learned change
Policy and chat template	Define the instructions and message boundary
Schema and validator	Define what the application accepts
Decoder configuration	Controls generation behavior and limits
Data and evaluator hashes	Bind the quality claim to a specific test
Runtime and hardware record	Scope performance and compatibility results

A content hash identifies file bytes, allowing a later check to detect whether an artifact changed after it was packaged. Establishing who created or approved those bytes requires a separate trust mechanism — a controlled registry and signing policy.

Inspect the manifest and active pointer

A manifest is a small record describing a release, including the files it contains and the metrics used by its gate. The companion’s make_release builds that record as a Python dictionary, hashing each named file with SHA-256 to identify its bytes.

The helper’s active pointer, active.json, stores a copy of the selected manifest so another process can discover the chosen release. Calling activate validates the candidate before writing this pointer — it does not load a model or reload a running worker.

The following activity creates this tiny release directory, where a text file stands in for the artifact we want to track:

temporary-release/
  candidate.txt
  active.json

Save the activity as inspect_release.py in the companion directory and run it with python3 inspect_release.py; the helper needs only the standard library. The temporary directory and its files are removed automatically when the with block ends, keeping the rehearsal separate from any real service.

import json
import tempfile
from pathlib import Path
from operations import make_release, activate

with tempfile.TemporaryDirectory() as directory:
    root = Path(directory)
    (root / "candidate.txt").write_text("candidate bytes", encoding="utf-8")
    manifest = make_release(root, "candidate", ["candidate.txt"], 0.95, 150)
    print(json.dumps(manifest, indent=2))

    pointer = root / "active.json"
    activate(root, manifest, pointer)
    selected = json.loads(pointer.read_text())
    print("Selected release:", selected["name"])
    print("Pointer contains the manifest:", selected == manifest)

{
  "name": "candidate",
  "files": {
    "candidate.txt": "732d058fadd90c70f22429227ab5d9c74919217099efe737aa46835ce3a60856"
  },
  "quality": 0.95,
  "latency_ms": 150
}
Selected release: candidate
Pointer contains the manifest: True

The files entry connects a path relative to the release root with the digest of that file’s contents. The supplied quality of 0.95 and latency of 150 milliseconds clear the helper’s default gates of 0.9 and 20,000 milliseconds. These numbers are rehearsal inputs — the trusted evaluation process belongs to the release system described in the next section.

Inside activate, check_release rejects an empty manifest, changed hashes, paths outside the release root, and invalid or out-of-range metrics. The final two printed lines establish which manifest the pointer contains, leaving model compatibility and loading to separate checks.

Bind the evaluator to the candidate it judged

Suppose an evaluation report says candidate B passed, but the proposed release accidentally contains candidate C’s adapter with its own valid hash. Detecting that mismatch requires the report to identify the evaluated adapter, then compare its digest with the proposed deployment.

The same relationship should cover the policy, decoder, validator, dataset, and evaluator, since changing any of them can change the score’s meaning. Retaining the test’s content hash and sampling description also makes one quiet substitution reviewable — replacing a difficult reserve with an easier one. A trusted promotion service verifies these relationships and who approved the report before accepting its quality claim.

The local helper’s file and threshold checks are building blocks within that system, with access to promotion controlled separately.

Files must remain unchanged between checking and loading, or a worker could serve different bytes from those the gate inspected. An immutable candidate directory prevents later writes; content-addressed storage identifies an object by a digest of its contents. With either approach, the selected artifact should be complete before promotion and protected against replacement after validation.

The control plane is the part of the system that chooses releases and coordinates deployment, while a worker loads the model and answers requests. The control plane confirms the candidate passed its gate; each worker confirms it loaded that candidate and handled a known test request. Record the worker’s actual release identity in response traces so an incorrect invoice can be traced back to the version that produced it.

Separate candidate creation from promotion

Training creates a fresh candidate directory, and evaluation reads that candidate before writing a separate report for the promotion decision. A failed run or rejected evaluation therefore leaves the selected release alone — and several candidates can be compared without changing live behavior.

The companion’s experiment runners refuse to reuse an output directory, keeping a new run separate from an earlier artifact. Its release helper writes the selected manifest to a temporary file beside active.json, flushes it, and replaces the pointer with os.replace. On a filesystem supporting atomic replacement, readers see the complete old or new pointer instead of partially written JSON. Keeping both files on the same filesystem is necessary for this local replacement step.

Figure 20.1 separates the helper’s selection decision from the serving work that follows it in a larger deployment system. A rejected candidate leaves the pointer unchanged; a successful replacement selects the candidate while running workers may still hold the previous model.

Figure 20.1. Validation precedes pointer replacement, so rejection leaves the selected release unchanged. The dashed handoff is work for a larger serving system: the local helper does not reload a worker. Loading and confirming the actual release identity are separate from selecting it.

Several promotion jobs also need coordination because an older candidate could otherwise overwrite a newer selection by finishing last. One option is compare-and-swap: replace the pointer only if its current value still matches the version the job expected. The local helper has no such concurrency control; a production registry must define ordering or locking and retain who initiated each change.

Rehearse promotion, rollback, and rejection

Rollback selects a previously accepted release through the same validation path used to promote a new candidate. The next activity makes the sequence visible as old, new, old, then attempts to promote a candidate whose file changed after hashing.

Save it as rehearse_release.py in the companion directory and run it with python3 rehearse_release.py. Like the first activity, it uses disposable text artifacts and changes only a temporary directory created for this run.

import json
import tempfile
from pathlib import Path
from operations import make_release, activate

with tempfile.TemporaryDirectory() as directory:
    root = Path(directory)
    (root / "old.txt").write_text("old bytes", encoding="utf-8")
    (root / "new.txt").write_text("new bytes", encoding="utf-8")
    old = make_release(root, "old", ["old.txt"], 1.0, 100)
    new = make_release(root, "new", ["new.txt"], 0.95, 150)
    pointer = root / "active.json"

    for release in (old, new, old):
        activate(root, release, pointer)
        print("Active:", json.loads(pointer.read_text())["name"])

    before = pointer.read_bytes()
    (root / "new.txt").write_text("changed after hashing", encoding="utf-8")
    try:
        activate(root, new, pointer)
    except ValueError as error:
        print("Rejected:", error)
    else:
        raise AssertionError("Changed artifact was accepted")

    assert pointer.read_bytes() == before
    print("Pointer unchanged:", pointer.read_bytes() == before)
    print("Still active:", json.loads(pointer.read_text())["name"])

Active: old
Active: new
Active: old
Rejected: hash mismatch: new.txt
Pointer unchanged: True
Still active: old

The exception explains why promotion stopped — the byte comparison proves it stopped before modifying the active pointer. The companion’s release test additionally checks a failed quality gate and a path outside the release root, covering two other rejection reasons.

A text artifact makes the selection mechanism easy to inspect without downloading or loading a model for this exercise. A real adapter bundle also needs compatibility and loading checks, including the tokenizer revision and the runtime’s ability to allocate it.

Make promotion gates reflect actual consequences

An aggregate quality threshold can hide a regression on a rare but expensive boundary even while the average score improves. The extractor’s ambiguous totals and the router’s source-precedence cases therefore need explicit regression expectations alongside the overall acceptance score.

A practical gate combines minimum quality, named-slice regressions, invalid-output limits, and a latency budget tied to the service target. Wrong non-null extraction values may deserve their own cap, while SQL needs both execution correctness and allowed-query behavior checked separately.

On a twenty-four-case SQL test, one answer changes accuracy by about 4.17 percentage points, making nearby thresholds sensitive to individual cases. Counts and per-request comparisons reveal that sensitivity and help identify which parts of the evaluation need more coverage.

Repeated seeds can help investigate stochastic training, but broader cases are still needed when the test misses difficult task boundaries. A missing measurement also needs an explicit failure state — a crashed evaluator or failed latency run must never default to a passing value.

Roll out to a controlled fraction

A canary release sends a small, defined share of traffic to a candidate while the rest stays on the current version. This limits exposure while providing an opportunity to observe the candidate under the service’s actual operating conditions.

The companion hashes an identifier into one of ten thousand buckets and compares that bucket with the chosen percentage threshold. Repeating the same identifier at the same percentage produces the same assignment — retries and investigation become predictable.

Save this activity as inspect_canary.py in the companion directory and run it with python3 inspect_canary.py:

from operations import canary

ids = [str(i) for i in range(1000)]
assigned = [canary(request_id, 10) for request_id in ids]
repeated = [canary(request_id, 10) for request_id in ids]
print("Same assignments on retry:", assigned == repeated)
print("First 10 identifiers assigned:", sum(assigned[:10]))
print(f"Full sample assigned: {sum(assigned)}/{len(ids)}")
print(f"Observed allocation: {100 * sum(assigned) / len(ids):.1f}%")

Same assignments on retry: True
First 10 identifiers assigned: 0
Full sample assigned: 116/1000
Observed allocation: 11.6%

The ten-percent setting defines a target across many identifiers — the printed counts show what this particular sample actually receives. Changing range(1000) to range(10) makes the small-sample difference especially visible, so monitoring should report observed allocation alongside the target.

Assignment also needs a deliberate unit: request-level routing can send one customer’s related requests to different model versions. A stable user or tenant identifier may preserve the consistency the application needs, provided the application chooses it independently of model output.

Candidate and control traffic should be comparable, since a canary receiving only easy requests could look better for reasons unrelated to quality. Regional or other deliberate slices can be useful when their purpose and comparison boundaries are made explicit.

Keep rollback ready as a normal operation

Rollback requires the prior base revision, tokenizer, prompt, validator, and configuration to remain available alongside the previous adapter. A schema migration can make the old output unacceptable — restoring weights alone may leave the original behavior unreachable.

Clear triggers make rollback usable during an incident, including severe field regressions, sustained latency breaches, or unexpected authorization-path failures. Some justify immediate reversal; others call for investigation while traffic remains limited to the canary allocation.

Preserve candidate predictions, error categories, deployment timing, and active-version history so reversal retains the evidence needed to explain what happened. In a real service, verify which release each worker serves, how caches and in-flight requests behave, and whether downstream consumers remain compatible.

A retry that crosses a rollback must also avoid importing an invoice or initiating an action twice. Chapter 18’s idempotency requirement still applies here: request identity and application transactions handle duplicate effects while release identity explains model behavior.

Turn traffic into data without turning mistakes into labels

Production logs contain useful examples whose labels still need review: an accepted answer can be wrong, and an edit may express a preference. A curation pipeline should retain the input, candidate output, correction, reviewer decision, and policy version that together explain the proposed label.

Confirmed labels can enter supervised training, while unresolved cases remain available for policy discussion without being treated as established answers. Mix targeted repairs with representative successful cases so the next adapter preserves ordinary behavior as it learns difficult boundaries.

Privacy and retention decisions belong before traffic is copied into a corpus, since useful examples may contain customer or supplier information. Data access and reuse policies determine which records can be retained and who can review or train on them.

Synthetic replacements can reduce exposure when they preserve the decision that made the example useful in the first place. Replacing every entity with one placeholder can remove the ambiguity the model needs to learn, making de-identification part of dataset design.

Decide whether the failure calls for retraining

A changed database column name usually calls for a schema-context repair, while a newly supported currency may require a validator and policy update. A missing source document needs retrieval or clarification, so the operational symptom should lead to a diagnosis before another training run.

Retraining becomes more justified when the task contract is stable, failures recur, and reliable labels represent the needed behavior. Compare simpler interventions where practical, especially when a prompt or retrieval correction could meet the same requirement with less release work.

Rising latency with unchanged predictions may come from workload growth, long documents, memory pressure, or a change in serving configuration. Measure the stage that slowed down before adding training examples — they are unlikely to repair those causes.

Disagreement among reviewers calls for a policy decision before their corrections become labels for the next specialist. Resolve and version the rule, then identify earlier examples whose labels must change to keep the training set consistent.

Keeping the current model is reasonable when a tiny gain on a narrow test fails to justify the migration and support burden — the evidence required for changing the service should grow with the consequences of making that change.

Reproduce a release from its recorded inputs

Reproduction begins with exact revisions and content hashes, alongside the pinned packages, seed, backend, and decoding settings used for a run. These records support investigation across machines, although differences in hardware and runtime can prevent bitwise equality.

A useful reproduction check asks whether the artifact loads, the task contract holds, and important predictions stay within accepted behavior. Systems requiring stricter numerical reproducibility need that requirement specified before choosing the infrastructure and the tolerances used in testing.

Raw predictions help explain differences that a summary percentage hides, including parsing errors, schema changes, and changes in the generated answer. Preserve enough input identity to connect each prediction to its labeled case while limiting unnecessary copies of sensitive text.

Retain the known-good release and the evidence needed to rebuild it, particularly private datasets or policy files with no other preserved copy. Large base caches can be removed when their exact revisions remain recoverable from another reliable store.

The public companion excludes weights and environments from Git, retaining scripts, pins, fixtures, and raw results to reconstruct the teaching runs. A private specialist also needs an artifact store whose retention and access policy covers the larger files used by its releases.

Where the work can go next

A broader, independently labeled evaluation corpus drawn from the intended workload is the most useful next investment for this bookshop system. Establishing whether the benchmark represents real decisions provides a stronger basis for further work than immediately choosing a more elaborate training method.

Distillation transfers a capable teacher’s behavior to a student, becoming useful when the teacher reliably solves the task under a clear evaluation contract. Teacher outputs still need filtering and provenance, with an independent test deciding whether the student has learned useful behavior.

Multilingual extraction, longer documents, schema retrieval for SQL, constrained decoding, and efficient adapter serving each offer further directions. Each extension changes the system enough to need its own baseline and failure analysis, using the measurement discipline developed throughout the book.

Across these twenty chapters, specialists have been useful where tasks are defined, alternatives are measured, and applications retain authority over model output. That gives us a practical basis for choosing and operating a specialist, including knowing when its current evidence supports keeping the existing system.

Exercises

1. The unchanged adapter

An adapter file is identical, but the chat template changes. Is it the same release behavior?

Answer

Not necessarily. The template changes the sequence presented to the model. Bind it to the release identity and rerun the relevant evaluation. An adapter hash alone cannot establish behavioral equivalence.

2. A rejected promotion

A candidate fails its hash check. What must the promotion test inspect besides the raised exception?

Answer

It must confirm that the active version remains unchanged. A gate that reports failure after updating the pointer has not protected the service. The companion’s test checks this state explicitly.

3. A stable canary

Why hash a stable identifier instead of drawing a new random number on every retry?

Answer

Stable assignment makes the same unit return to the same version, which helps consistency and investigation. Choose the unit deliberately: request, user, or tenant assignment has different consequences. Stable hashing alone does not guarantee representative traffic or distributed rollout safety.

4. A rising null rate

The extractor begins returning more null dates after a supplier changes its invoice format. Should a retraining job start automatically?

Answer

Investigate first. The source may now contain ambiguous dates, OCR may have dropped information, or the policy may need an explicit locale rule. Retrain only after identifying the cause and obtaining reliable labels for the intended behavior.

Final thoughts

A production specialist needs behavior we can explain, measure, and reverse. Keep its task contract close to the data and its evidence attached to the release. Permission and consequence belong in the application — so we can improve the model while retaining our own definition of “good enough.”