Failure-Driven Fine-Tuning

A training run leaves us a score and a set of wrong answers. The score tells us whether the bookshop’s candidate meets its requirement, while the individual failures help decide what to change next. Adding rows is useful only after we identify which decisions those rows should teach.

An extractor that guesses ambiguous dates needs examples distinguishing uncertainty from a supported date. A router needs to distinguish quoted instructions from user intent, and a SQL model may need records separated from quantities — the repair depends on the decision that failed.

Failure-driven fine-tuning uses observed mistakes to choose the next training examples, then checks whether the new candidate improves. This study starts from saved failures of a clean invoice extractor: the original adapter supplies the diagnosis, and two fresh adapters test the proposed repair. The repairs come from a programmatically prepared pool, so this experiment does not call Chapter 11’s teacher or continue one of its students.

Run against: Python 3.13.3, Transformers 4.56.2, PEFT 0.17.1, PyTorch 2.8.0; Qwen3-0.6B float32 LoRA on Apple M1 Ultra/MPS, failure_study.py, seed 42, matched sixty-step control and repair runs.

Not run. Independent human labeling, a new real-document distribution, multiple repair rounds, multi-seed uncertainty, and causal isolation of every data-selection effect are outside this experiment.

The fresh audit changes record identities and values while retaining the known perturbation families. It is a new synthetic audit sample, not an independent generalization benchmark.

Turn an error into a reproducible case

A reproducible failure record binds the input identity, task-policy version, model and adapter identity, raw output, expected answer, and evaluator’s reason. Without them, an investigation can quietly shift to a different prompt or model under the same informal description.

The raw answer explains what the model did before any serving cleanup changed it: stripping Markdown can hide a formatting failure, and converting a literal "null" into JSON null can hide a semantic error. Such repairs may belong in the serving system, but diagnosis needs the original output too.

The invoice contract is the one introduced in Chapter 11: six fields, with strings for supported facts and null for missing or unresolved facts, and the gold record is the known answer key. A structurally valid answer can still copy the wrong value; an exact answer must match the whole record. Chapter 13 develops the original extractor training, while the saved predictions let us inspect its failures here.

The initial clean adapter learned from tidy examples, then faced eight perturbation families: missing terms, ambiguous currency, ambiguous dates, conflicting totals, multiple entities, noise, typos, and format changes. Its challenge predictions now supply development evidence — the mistakes that guide this repair study.

That change of role is explicit: once the challenge errors guide training, the challenge set is no longer an untouched final test for the repaired candidate. It remains a useful regression and diagnostic set, but a fresh audit is needed for the next decision — cases whose answers have not guided this repair.

The saved case extract-challenge-000 contains no payment terms:

Invoice INV-1600. Vendor: Challenge Paper 600. Total: USD 700.00.  Date: 2026-01-01.

The original clean adapter emits "terms":"NET_30" where the answer key contains "terms":null; the other fields match, so the record is structurally valid but not exact. This is a specific failure to preserve a missing fact. A contrasting repair should include both an invoice with no terms and one with explicit terms, so the model has to attend to the source.

Categorize failures by the decision that broke

“Wrong answer” gives us little basis for choosing training data, whereas “copied buyer instead of vendor” suggests role-disambiguation examples. A guessed ambiguous date suggests abstention examples, prose around JSON suggests output-contract examples or a serving constraint, and in each case the useful category identifies a distinction we can act on.

A record may need tags at several levels: structure describes the output contract, fields identify wrong facts, and behavior suggests a possible explanation. Keeping those levels separate lets us investigate a hypothesis without treating it as an observed cause.

Observation	Likely missing distinction	Candidate repair
Buyer copied as vendor	Entity role versus prominence	Swap positions while preserving roles
Date guessed from 03/04/26	Ambiguity versus normalization	Explicit null and unambiguous-date pairs
First of two totals copied	Conflict versus precedence	Conflicting and explicitly corrected totals
Footer changes the answer	Document content versus instruction	Source-preserving footer variants
Correct facts in invalid JSON	Facts versus output contract	Exact schema examples and validation

“Likely” marks a hypothesis about the failure — not an observation of the model’s internal reasoning. A counterfactual example changes a relevant feature while holding the others steady, letting us test whether the proposed distinction affects the answer.

Move the buyer above and below the vendor while keeping the role labels clear. Copying the last company in both arrangements makes position a plausible contributor, while failing only on an unfamiliar label points toward vocabulary; the repair should follow the distinction those cases actually expose.

Prioritize errors by impact and recurrence

A frequent harmless omission and a rare costly hallucination need different responses. The application must distinguish failures that block automation from those it can tolerate temporarily or send for review, so that repair effort follows consequences as well as counts.

For invoices, a wrong non-null amount may be more consequential than a missing optional term, and for routing, sending a request to an unauthorized data path is different from sending it to clarification. For SQL, a valid but semantically wrong query may be more dangerous than an obvious syntax rejection.

The synthetic challenge set gives every perturbation family equal weight, which helps diagnosis without estimating the bookshop’s traffic distribution. Estimating production prevalence would require a representative sample, so these family counts cannot serve both purposes.

A repair queue can combine observed frequency, consequence, label confidence, and expected repair cost. Uncertain labels often need policy clarification first — further training cannot make an unresolved business rule into a defensible answer key.

Ordinary successful cases define behavior the repair should preserve, even when dramatic failures draw most of our attention. Retaining them in training and regression evaluation lets us detect a candidate that improves the targeted behavior while damaging routine work.

Freeze the repair rule before reading the audit

The companion selects the perturbation kinds for which the clean adapter has at least one incorrect challenge record, then takes the available repair examples from those kinds. The rule is simple and recorded in runs/failure-v1/plan.json.

The rule is deliberately coarse — a failed example can select an entire family, without ranking its cost or distinguishing subtypes. That simplicity makes each repair row traceable to a recorded selection decision, though it does not make the allocation optimal.

The repair pool uses different record values from the challenge set, so selection does not copy failed test documents into training. The shared construction families still mean the new rows are distinct instances rather than unfamiliar kinds of case.

The plan records the selected kinds, repair IDs, development source, step budget, and audit scope before the audit predictions are evaluated. That ordering means the audit’s failures cannot select the repair that is then judged on the same audit.

A real curation process should preserve this order even when generation and review take days. The reserve can be created or sampled under a documented policy, sealed from the training decision, and evaluated after the candidate is fixed. What makes it a reserve is its role in the decision — not the filename test.jsonl.

Mix repairs with replay examples

Training only on failures changes the task distribution the optimizer sees: repeated ambiguous-date invoices could teach blanket null dates, while repeated mixed-source router repairs could neglect ordinary single-source requests. The new collection must retain the situations where the existing behavior was useful.

Replay examples are previously useful cases included with the repairs, so training still encounters behavior the new candidate should preserve. The companion retains its 120 clean extraction records for that purpose alongside examples of the missing distinctions.

The repaired arm combines those clean records with the selected messy records, and the control arm uses only the clean records. Both start from the same original base checkpoint with fresh adapters, rather than continuing from the adapter that supplied the development failures.

Starting fresh keeps the training histories simpler to compare. Continuing from the old adapter would be a valid operational experiment, but it would add inherited optimization history, and the question would become whether continued training improves that particular checkpoint.

The deterministic shuffle fixes the training order, but a larger pool changes exposure under a fixed-step budget. Replay gives old cases a place in that order — it does not guarantee that every old behavior survives training.

Follow the selected repair categories

The clean adapter misses all four cases in each of four challenge families — missing terms, ambiguous currency, ambiguous date, and conflicting total — and two of the four multiple-entity cases. The noise, typo, and format families have no incorrect records in this particular development result.

The selection rule therefore chooses five families, and since the repair pool contains eight examples per family, forty messy rows are added to the 120 clean replay rows. That is a consequence of the rule, not a mixture adjusted after reading the fresh audit.

The zero-error families are not declared solved — they simply do not trigger this repair round, and a different development sample could expose errors in them. Keeping their cases in regression evaluation helps detect a candidate that damages behavior that currently works.

The all-or-nothing family rule also has a cost: multiple entities contributes eight repair rows despite only two observed failures, while a fully failed ambiguity family also contributes eight. That is a transparent equal-family allocation rather than a risk-weighted optimum, and a production process might allocate more review effort to a costly amount error than to a missing optional term.

Add a matched training control

Extra optimizer steps can improve a model even when the proposed repair data contributes little. The companion therefore gives the clean-only control and the repair mixture sixty steps each at batch size two, with seed, learning rate, rank, alpha, and decoder also matched.

Sixty updates at batch size two give each arm 120 example presentations. The control sees its 120 clean rows once, while the repair pool contains 160 rows, so the fixed budget visits only a deterministic shuffled selection of 120 and leaves forty unpresented. Pool size describes what is available, not what reaches the optimizer — the saved pool IDs, seed, and training code make the order reproducible.

Figure 12.1. Read upward. The original clean adapter supplies the failures used to select repairs, but its weights do not pass into either arm. Both candidates start fresh and receive the same update budget. The repair pool is larger; sixty batches of two present only 120 of its 160 rows. The shared evaluation sets keep the resulting predictions comparable.

This differs from the later fixed-epoch augmentation study, where adding records also adds steps. Both designs are useful, but the matched-step comparison asks the narrower question of whether this selected mixture helps under the same number of updates.

The control removes the extra-optimization ambiguity while example content, coverage, and repetition frequency still differ — random augmentation, matched-size clean data, and several seeds could separate those effects further.

Inspect the plan before running a repair

The saved plan lets you see what selected the data without training a model. From the companion directory, save this as inspect_repair.py and run python3 inspect_repair.py; it uses only the standard library:

import json
from pathlib import Path

root = Path("runs/failure-v1")
plan = json.loads((root / "plan.json").read_text())
print("source:", plan["development_run"])
print("families:", ", ".join(plan["failed_kinds"]))
print("repair rows:", len(plan["repair_ids"]))
print("steps:", plan["training_steps"])

source: extractor-clean-v1/challenge-zero.jsonl
families: ambiguous_currency, ambiguous_date, conflict, missing, multiple_entities
repair rows: 40
steps: 60

The source line identifies the original diagnostic run, and missing connects our payment-terms failure to a selected family. The repair IDs name the available pool; the fixed-step sampler determines which rows are actually presented. The two trained candidates use fresh adapters, as Figure 12.1 shows.

To reproduce the model work, use the companion’s frozen training environment on the supported MPS setup. First create the clean extractor and its challenge predictions, then give the repair study a separate output prefix:

.venv/bin/python specialist_run.py \
  --task extractor --train --splits test challenge \
  --output runs/local-extractor-clean
.venv/bin/python failure_study.py \
  --source-run local-extractor-clean --prefix local-failure

Both commands require new output directories. The first supplies the diagnostic predictions; the second writes runs/local-failure-audit-v1/plan.json, trains the control and repair candidates, and stores their audit predictions there. Their model artifacts live in runs/local-failure-failure-control-v1/ and runs/local-failure-failure-repair-v1/, where the repeated word comes from the prefix plus the arm name, and local prefixes preserve the published results and data/extractor/audit.jsonl.

Make the fresh audit new in a stated way

The audit contains thirty-two records with new IDs and modified invoice references, vendor names, and amounts. It retains the eight known perturbation families. The construction changes the supported fields and source text together so the labels remain aligned.

The invoice references gain an AUD- prefix, and vendor and amount values differ from the development cases, so the audit measures transfer to new values rather than another encounter with the same documents. Two facts belong together when reading it. Its answers did not choose the repair, which is what makes it fresh; its templates are the familiar eight families, which is what keeps it from being independent. Unseen layouts, new languages, independently authored wording, or new kinds of ambiguity would need a broader reserve.

Before reading the table, decide what would count as a useful repair: it should improve exact answers on the audit while retaining clean answers. If exact matches improve but validity stays unchanged, some accepted records have better values while the structural failures remain, and the paired predictions below let us check which cases changed.

Selected development failure families: ambiguous_currency, ambiguous_date, conflict, missing, multiple_entities.

Measure	Clean control	Targeted repair
Training rows	120	160
Steps	60	60
Clean exact	24/24	24/24
Development exact	14/32	23/32
Audit exact	8/32	15/32
Audit valid	17/32	17/32

Check the target improvement and the regressions

The matched repair arm preserves 24/24 clean exact matches and raises development exact matches from 14/32 to 23/32. Nine cases change from wrong to right and none from right to wrong, with four gains in missing terms, three in ambiguous currency, and two in multiple entities. Ambiguous dates and conflicting totals still have no exact matches in either matched-step arm.

This is a selective repair, not completion of the challenge task. The larger fixed-epoch augmentation experiment in Chapter 14 reaches 32/32 under a different training budget and mixture, and its result must not be substituted for this sixty-step comparison. Keeping both visible distinguishes a useful improvement from a claim that one small repair round solved every failure.

On the fresh audit, exact matches rise from 8/32 to 15/32 — seven wrong-to-right changes and no right-to-wrong changes — while both arms still produce only 17/32 structurally valid records. Wrong non-null values fall from nine to two, and the strict evaluator counts eighty missing non-null fields in both arms. That count includes whole records rejected by the contract, not eighty literal nulls emitted by the model.

The new values expose another weakness. For extract-audit-000, both arms emit currency AUD, apparently confusing the new AUD-INV-1600 reference prefix with a currency cue; the source currency remains USD, and AUD is outside the contract. The repaired arm correctly changes the missing payment terms to null — but the currency error still invalidates the whole record. That reading is a plausible interpretation of the output rather than a proven account of the model’s reasoning. The audit as a whole supports a limited improvement, while its low absolute accuracy argues against promotion.

Whole-record exact match reflects an import that can fail because of a wrong field; per-field accuracy helps locate that field. Missing non-null fields and incorrect non-null values then distinguish withheld facts from unsupported completion, subject to the strict validation gate.

The net gain of seven on the audit could in principle have been ten repairs and three regressions. Counting each direction separately is what shows it was seven and none, and keeping the case IDs lets the application judge any case that does regress. The activity below makes that comparison on the saved audit.

On a thirty-two-case audit, one record changes the score by 3.125 percentage points — extra decimal places add no evidence. Preserve exact counts in the result files and use readable percentages in the chapter when they help interpret those counts.

A lower training loss shows that the optimizer fits the repair mixture better, which is useful during training. Promotion depends on reserve and regression outcomes, because they test behavior beyond those training presentations.

Count repaired cases and regressions

A higher total can conceal damage to previously correct answers, so this standard-library activity pairs the saved audit predictions by ID. Save it as compare_repairs.py in the companion directory and run python3 compare_repairs.py:

import json
from pathlib import Path

root = Path("runs/failure-v1")
def load(name):
    rows = [json.loads(line) for line in
            (root / name).read_text().splitlines()]
    return {row["id"]: row for row in rows}

control = load("clean-audit.jsonl")
repair = load("repair-audit.jsonl")
assert control.keys() == repair.keys()
gains = losses = 0
for identity, before in control.items():
    after = repair[identity]
    gains += not before["exact"] and after["exact"]
    losses += before["exact"] and not after["exact"]
print("wrong to right:", gains)
print("right to wrong:", losses)
for name, rows in [("control", control), ("repair", repair)]:
    row = rows["extract-audit-000"]
    answer = json.loads(row["raw"])
    print(name, answer["currency"], answer["terms"],
          row["valid"], row["exact"], row["missing"])

wrong to right: 7
right to wrong: 0
control AUD NET_30 False False 5
repair AUD None False False 5

The last two lines show currency, terms, validity, exactness, and the missing-field count for one case. Python prints None for JSON null — payment terms improved, but the unsupported AUD currency still rejects the record. The source has five non-null target fields and the strict evaluator credits none when the contract fails, which is why the missing count is five even though the raw answer visibly contains several correct strings. In the aggregate table, “missing” therefore includes facts lost at the validation gate as well as literal omissions.

Resist the temptation to repair the audit immediately

After inspection, the audit’s failures can guide the next development round, but that use changes its role. It can remain regression evidence, but no longer act as a fresh final reserve for a repair chosen from its mistakes.

A growing regression collection preserves known boundaries, while a rotating or newly sampled decision reserve tests cases that did not directly shape the candidate. Keeping those roles separate lets the team retain useful failures without treating familiarity as a new estimate of generalization.

Repeatedly optimizing against a public benchmark can also create indirect leakage: even without copying test text into training, engineers learn which families and phrasings matter. That knowledge changes prompts, generation matrices, and model selection.

These fixtures are public teaching data, so once published they are especially unsuitable as a permanent independent benchmark for future claims. You can still use them to learn the workflow, check an implementation, and reproduce the reported comparisons.

For a product, refresh the reserve from the intended traffic under a documented sampling and review process. Keep enough historical cases to catch regressions, but do not confuse regression protection with a fresh estimate of generalization.

Use a teacher to target the gap, not to repeat the failure

“Generate more examples like this wrong answer” can reproduce the mistake we are trying to remove. A useful teacher prompt instead explains why the correct label follows from the source and identifies the feature that should vary.

For an ambiguous-date repair, pair explicit year-month-day dates with ambiguous local forms, and for entity roles, vary names, order, and prominence while preserving vendor versus buyer labels. For conflicting totals, distinguish unresolved conflict from an explicit correction.

The generated target should follow the policy and the candidate should pass task-specific validation: known source records for extraction, executable oracles for SQL, careful source-policy review for routing. The teacher’s confidence is not an answer key.

Our repair pool has known fields and a traceable selection rule, and Chapter 11’s teacher could supply additional candidate text provided each rendering preserves the intended answer. Testing that combination would be another experiment; the result here belongs to the programmatic pool.

A small, well-checked repair set can teach a missing distinction while replay helps preserve established behavior. More wording around an already easy decision may consume the same training budget without addressing the observed error.

Stop when another layer owns the problem

Some failures should not enter the training queue. An unauthorized request belongs to access control, a missing attachment to input validation, and a stale inventory value to the source path; a validator rejecting a newly approved schema belongs to a coordinated contract update.

Training can hide symptoms without addressing their cause — guessing missing fields or avoiding a query pattern may conceal an input or schema problem. A narrow score can improve while the system becomes harder to diagnose, which is why the layer responsible matters.

Before deciding to change the weights, ask whether the model had enough information, whether the expected answer was unambiguous, and whether the application supplied the right tools and permissions. Those questions distinguish a training problem from a failure elsewhere in the system.

Another training round may add little once the candidate meets the requirement and review safely handles the remaining cases. Repeatedly trading errors between families instead calls for reconsidering task scope, model capacity, or evaluation design.

Chapter 20 adds release gates and rollback to this loop, which make improvement operationally reversible. The data discipline comes first: a reversible deployment cannot rescue a dataset whose labels do not express the intended task.

Exercises

1. Choose a repair and preserve a success

For extract-challenge-000, name the failed decision and propose a repair example with different invoice values. Then choose a contrasting clean example to retain as replay. Both candidates will receive sixty steps at batch size two. Explain why the experiment should start both with fresh adapters.

Answer

The model supplied payment terms absent from the document. A suitable repair changes the invoice identity, vendor, and amount while still omitting terms; its target terms value is null. Replay should include an invoice that explicitly states terms, with that value in the target. Together they distinguish absence from presence. Fresh adapters and the matched step budget reduce differences in inherited training history and extra optimization; they do not guarantee that data coverage and exposure are otherwise identical.

2. The reused test

A team reads test failures, generates targeted examples, and reports the improved score on the same test as fresh generalization. What is wrong?

Answer

The test guided the repair and now serves as development or regression evidence. Use a new reserve for the next generalization decision, while retaining the old cases to ensure the known failures stay fixed.

3. A cautious regression

The repaired extractor emits null for every date. It handles all ambiguous-date cases correctly. Has the date behavior improved?

Answer

Only on that subset. Check explicit dates and count missing non-null fields. A useful repair learns when to abstain, not a blanket rule to discard the field. Replay and clean regression cases expose this failure.

4. Read the paired audit

Use compare_repairs.py to count gains and regressions. Then explain why fixing the payment terms in extract-audit-000 does not create an additional exact match. What does an audit with new values but familiar perturbation families establish?

Answer

The saved audit has seven wrong-to-right changes and no right-to-wrong changes. For extract-audit-000, both candidates still emit unsupported currency AUD, so the record remains invalid even after terms become null. The audit tests transfer to new values within known families under this training plan; it does not establish robustness to independently authored documents or unseen perturbations.

Final thoughts

A failure is valuable when it becomes a precise, labeled decision the next candidate can learn. Keep the repair rule visible, retain a control, preserve ordinary behavior, and give the audit a role that training has not already consumed. The loop then produces knowledge about the system instead of a sequence of improving scores.

Next: From Classification to Structured Generation