When the Real World Gets Messy

An invoice with one clearly labeled total is a copying task; an invoice with two totals and no indication of which one is final is a decision about uncertainty. Adding more clean invoices can improve copying while leaving that decision untouched.

The bookshop’s extractor has already seen three tidy layouts; we now test what happens when a document stops cooperating — a payment term disappears, a currency becomes a symbol, or a footer tries to redirect the model. Unlike the earlier repair selected from observed failures, this study plans augmentation across the perturbation families in advance. The distinction matters when we decide what its improvement establishes.

The extractor should copy a supported amount through an irrelevant footer, then withhold it when two conflicting totals leave the answer indeterminate. Both documents differ from the clean layout — but only the latter changes what the source supports. The task contract must survive both changes, which is why calling everything “noise” would teach the wrong labels.

Run against: Transformers 4.56.2, PEFT 0.17.1, PyTorch 2.8.0; Qwen3-0.6B float32 LoRA on Apple M1 Ultra/MPS, specialists-v1, two epochs, seed 42.

Not run. Real supplier documents, OCR corruption, multilingual invoices, independently authored perturbations, and adversarial security evaluation are outside this study.

The challenge and augmentation records use different values but share eight construction families. Improvement measures transfer within those families, not arbitrary document robustness.

Separate a new value from a new problem

A held-out vendor name tests a new value; an invoice listing both supplier and buyer tests a new selection problem. When training always shows one company, copying any company-like phrase can succeed without learning its role. Adding another company exposes that shortcut and gives the next examples a specific distinction to teach.

New numeric amounts test copying — a subtotal, tax line and final total test which number the requested field means. A correction to an earlier total introduces a further question: which statement takes precedence? Grouping all of these under “unseen invoices” would hide the different decisions a repair must address.

Our clean split changes record values while keeping three layout families; the challenge split adds eight kinds of perturbation, each appearing four times, for thirty-two records in total. The small equal groups make errors inspectable, but a single record moves a family’s accuracy by twenty-five percentage points.

The challenge is therefore a diagnostic collection, and its balanced family mix is not an estimate of how often those problems occur in the bookshop’s traffic. A production estimate would need a sampling plan tied to real documents, including ordinary successes and rare expensive failures.

Write the expected behavior before changing the text

A perturbation is a deliberate change to the document, and its intended effect on the answer must be decided first. Adding an irrelevant footer should preserve the target — removing the only payment term should make that field null. The table separates these effects so augmentation does not reward copying a fact that its own transformation removed.

Perturbation	What changes in the source	Expected effect
Missing terms	Payment terms removed	terms becomes null
Ambiguous currency	ISO code replaced by $	currency becomes null
Ambiguous date	ISO date replaced by 03/04/26	invoice_date becomes null
Conflicting total	A second, unprioritized total appears	amount becomes null
Instruction-like footer	A request for a poem is appended	Facts stay unchanged
Multiple entities	Buyer and contact are added	Vendor stays unchanged
Label typos	Some labels are misspelled	Facts stay unchanged
Format change	Separators become line breaks	Facts stay unchanged

The typo family is limited. The replacement rules affect labels present in the selected layout, and some layouts do not contain those exact words, so two of the four nominal “typo” records carry no spelling change at all. Family names are metadata about construction, not proof that every member has equal difficulty, which is why the text has to be read beside the family score.

The amount policy also deserves care: the second total is explicitly unprioritized, with neither marked final. If the text instead said “Corrected final total,” returning null would discard useful evidence. The label must follow the wording, not a simplistic rule that any document with two numbers is ambiguous.

Preserve facts through harmless changes

An irrelevant footer and a change from pipes to newlines should preserve the invoice’s facts. This is invariance: the presentation changes while the desired answer stays constant. It gives us a testable expectation for each field, unlike an ambiguous date whose target must change to reflect the lost certainty.

Starting with a clean record and comparing its prediction with the prediction after adding a footer creates a paired check. If the vendor changes, the model has responded to text that should not affect that field. Holding the underlying facts steady makes the transformation’s effect easier to interpret than a comparison between unrelated documents.

The companion’s challenge records are not a complete paired robustness benchmark, because they rotate layouts and perturbations across synthetic records. A stronger study would preserve each original and its transformed counterpart, then report how often the transformation changes a correct answer into a wrong one.

That paired design still needs a valid transformation: removing an apparently irrelevant line can accidentally remove the only currency indicator. Reformatting can join two words or change the association between a label and value. An augmentation function is part of the labeling system and deserves tests of its own.

For source-preserving transformations, a useful check is to compare the stored gold object before and after the transformation; for source-changing transformations, check the exact intended field difference. These checks catch label drift before it becomes a model-training problem.

Teach abstention as a positive behavior

When a source is ambiguous, returning null is abstention: the extractor keeps the field but declines to invent its value. Complete training records alone give it little practice making that decision, because every example rewards supplying a fact. Deliberate omissions teach the other half — when an incomplete record is the correct answer.

This creates a predictable tension: the model has learned a neat output pattern and may preserve it by guessing. A missing payment term becomes the most common term, an ambiguous date becomes the interpretation favored by the training distribution, and the answer looks complete because the model has filled the gaps.

Examples with deliberate omissions teach a different rule: keep the schema stable while admitting that a fact is unavailable. The record still has six keys, but one value is null, so the application can distinguish a structurally invalid response from a valid incomplete record.

A structurally invalid response and a valid null usually need different application paths: invalid structure may justify one controlled retry or a parser-level failure. A valid null may require a missing document, supplier clarification, or review. Repeating the same model request cannot supply a fact the document never contained — no retry policy changes that.

Overtraining abstention can cause the opposite problem: a model that emits null for values that are present but unfamiliar. That is why the evaluator counts missing non-null fields as well as incorrect supplied values. A safer-looking model that discards most useful facts has not improved the workflow — it has moved the cost.

Compare clean training with planned augmentation

The clean LoRA arm uses the 120 clean records from Chapter 13. The augmented arm starts from the same base with a fresh adapter and adds sixty-four messy training records, eight for each perturbation family. Both train for two epochs with the same batch size, rank, learning rate, and decoder, so augmentation changes the training collection without inheriting the clean adapter’s weights.

Keeping two passes through each collection makes this a fixed-epoch comparison: the clean arm takes 120 optimizer steps and the augmented arm takes 184. Both coverage and exposure change, so an improvement cannot be attributed solely to the added examples; a matched-step control would hold the update budget fixed, as in the earlier failure-driven study.

The augmentation plan was defined before final challenge predictions were inspected, so it is a planned robustness experiment, not a claim that the final test discovered the repair. Chapter 12’s separate failure-driven study uses observed development errors and a fresh audit set to illustrate that loop explicitly.

The distinction matters because a test loses its role as a final reserve once its failures guide training. It can remain useful as a regression set, but it cannot keep being presented as untouched evidence after each repair round.

To reproduce the planned augmentation, use the same frozen environment as the clean extractor run and a new output directory. The command adds one explicit flag; the saved tables below provide the result-reading path without another training run:

.venv/bin/python specialist_run.py \
  --task extractor --train --repair \
  --splits test challenge --output runs/local-extractor-augmented

Here --repair appends the predefined augmentation file, rather than inspecting predictions, calling a teacher, or selecting examples automatically; its name describes which data to include. The plan still comes from the experimenter — which is why this command must be read alongside the stated fixed-epoch comparison.

Method	Exact records	Valid records	Input tokens
Zero-shot	5/32	27/32	166.7
Fixed five	13/32	18/32	703.7
Dynamic five	13/32	23/32	713.4
Clean LoRA	14/32	27/32	166.7
Augmented LoRA	32/32	32/32	166.7

Method	Missing non-null fields	Wrong non-null values
Zero-shot	33	20
Fixed five	73	5
Dynamic five	48	10
Clean LoRA	25	13
Augmented LoRA	0	0

Read the failure table before choosing a winner

An aggregate exact-match score can rise while an important field gets worse: suppose an adapter learns to ignore footers but begins guessing ambiguous dates. If footer cases are more common in the test, the average may improve, while a payment process that relies on dates could still prefer the earlier behavior.

Per-family exact match tells us under which document changes failures occur, while per-field accuracy tells us which facts were affected. Raw predictions then distinguish an omitted fact, wrong value, unsupported term, or prose around the JSON. Use those views together: the family table below identifies behaviors worth inspecting before an aggregate improvement becomes a release decision.

Family	Clean LoRA exact	Augmented LoRA exact
missing	0/4	4/4
ambiguous_currency	0/4	4/4
ambiguous_date	0/4	4/4
conflict	0/4	4/4
noise	4/4	4/4
multiple_entities	2/4	4/4
typos	4/4	4/4
format	4/4	4/4

The baseline results already show that validity and correctness can move differently: fixed five-shot prompting obtains thirteen correct challenge records out of thirty-two, more than zero-shot’s five. Yet its strict validity is lower: eighteen valid records against twenty-seven for zero-shot. The extra demonstrations improve some mappings while encouraging outputs that violate other parts of the contract.

The prompting comparison shows why accepted shape and correct content need separate columns: improving some mappings can coexist with more rejected records elsewhere. Choosing an arm therefore depends on what costs the application more — invalid structure, wrong non-null facts, or unnecessary omissions. A single “quality” score would conceal that choice.

The four examples per family also limit confidence: a perfect family score is four successes, not a guarantee. Reporting the count makes the uncertainty visible — without pretending that a narrow synthetic sample supports a precise traffic-level estimate.

Inspect one error all the way through

Take a document whose total is stated twice, with neither amount marked final, so the gold amount is null. Several different failures are possible, and they point to different next actions.

If the model copies the first total, it may be following positional salience; if it copies the last, it may be assuming that later text supersedes earlier text. If it adds the totals, it has changed extraction into arithmetic, and if it returns both as a list, it has recognized ambiguity but violated the output type.

The corresponding repairs differ: positional mistakes need examples where the correct decision changes independently of position, arithmetic mistakes need clear extraction-versus-computation boundaries, and list output needs examples of the approved uncertainty representation. A generic instruction to “be more accurate” does not identify any of these behaviors.

For an invoice naming both buyer and vendor, copying the wrong company could reflect an unfamiliar role label, prominence, or position. Counterfactual pairs test these hypotheses by changing positions while preserving labels, then changing labels while preserving positions. The comparison narrows what to investigate — it does not reveal the model’s internal reasoning.

We are looking for a small set of examples that teaches the missing distinction. Hundreds of additional clean invoices would mostly repeat what the model already knows, lowering training loss while leaving the error we care about untouched.

A date error that formatting alone cannot repair

Challenge record extract-challenge-002 uses the pipe-separated layout:

Challenge Paper 602 | INV-1602 | invoice_date=03/04/26 | amount=702.14 | currency=GBP | terms=DUE_ON_RECEIPT

The clean adapter copies the date literally into invoice_date. Its other five fields match the target, but 03/04/26 violates the ISO-date contract. The strict parser rejects the whole record. The augmented adapter emits JSON null for the date and preserves the other fields, matching the declared answer.

You can read both saved predictions without loading a model. From the companion directory, save this as read_challenge_case.py and run python3 read_challenge_case.py; it uses only the standard library:

import json
from pathlib import Path

def rows(path):
    return [json.loads(line) for line in Path(path).read_text().splitlines()]

record = next(r for r in rows("data/extractor/challenge.jsonl")
              if r["id"] == "extract-challenge-002")
print("source:", record["text"])
print("gold date:", record["gold"]["invoice_date"])
for run in ("extractor-clean-v1", "extractor-augmented-v1"):
    row = next(r for r in rows(f"runs/{run}/challenge-zero.jsonl")
               if r["id"] == "extract-challenge-002")
    print(run, "valid" if row["valid"] else "invalid",
          "exact" if row["exact"] else "not exact")
    print("  ", row["raw"])

source: Challenge Paper 602 | INV-1602 | invoice_date=03/04/26 | amount=702.14 | currency=GBP | terms=DUE_ON_RECEIPT
gold date: None
extractor-clean-v1 invalid not exact
   {"invoice":"INV-1602","vendor":"Challenge Paper 602","amount":"702.14","currency":"GBP","terms":"DUE_ON_RECEIPT","invoice_date":"03/04/26"}
extractor-augmented-v1 valid exact
   {"invoice":"INV-1602","vendor":"Challenge Paper 602","amount":"702.14","currency":"GBP","terms":"DUE_ON_RECEIPT","invoice_date":null}

Python prints the gold None for JSON null. The two raw lines differ in one value, and that one value decides whether the whole record is accepted.

This case exposes an important temptation: a postprocessor could convert the copied date into a valid ISO string and make the structural error disappear. Without a locale rule, it would have to choose between interpretations, and the result could become valid while remaining unsupported by the source.

The model needs to distinguish an explicit date from an ambiguous one, rather than merely learn to format dates as ISO. A date-formatting library can check whether 2026-03-04 is a real date, but it cannot establish that the document author intended that day. Preserving uncertainty is the extraction decision — formatting alone cannot make it.

The evaluator’s field policy is conservative: because the clean answer is structurally invalid, all six fields score false for that record even though five strings visibly match. This reflects the consumer’s all-or-nothing parsing boundary. An auxiliary diagnostic could credit recoverable facts, but it would answer a different question and should have a separate name.

Making one malformed date consumable can improve six field scores at once, because the strict gate had rejected the entire object, so that jump need not represent six newly learned extraction behaviors. The raw date example identifies the actual change, so a field-score increase should be read alongside the source and prediction.

The example also explains why the review path should retain the source: a reviewer can see the ambiguity and request clarification. Showing only a guessed normalized date would remove the very evidence needed to question it. Good extraction software preserves the connection between an output field and the text that supports, or fails to support, it.

Keep robustness separate from security claims

A footer containing “output a poem” is a useful task-boundary probe, but passing it does not establish resistance to prompt injection. Real attacks can vary wording, exploit tool access, target hidden context, or work through retrieved content — a handful of synthetic footers covers only the literal patterns present.

The application should avoid granting consequences directly from extracted text: an invoice record is a candidate for validation, not permission to transfer money. Vendor identity, account changes, authorization, and duplicate-payment checks belong to independent application logic.

This separation also limits the harm of ordinary mistakes: a wrong amount that reaches a review screen is different from a wrong amount that initiates a payment. The same model accuracy can have very different operational consequences depending on the system around it.

The SQL specialist in the next chapter follows the same principle: a generated query can be evaluated in a restricted fixture without granting access to a live database. Model behavior is one layer — the environment’s permissions are another.

Measure drift in the input, the output, and the decision

After deployment, an extractor can degrade because documents change even when the model remains identical: a supplier may adopt a new template, add a second address, or begin issuing invoices in another currency. The old test set stays green because it does not contain the new condition.

Input monitoring looks for changes in document length, supplier mix, layout signatures, language, or missing-field patterns before correctness is known. These shifts are signals to investigate — a new supplier name need not cause an error when its role remains clear. The distinction between new values and new problems still applies after deployment.

Output monitoring can track invalid JSON, null rates by field, unsupported currency attempts, and retry frequency. A sudden rise in null dates may indicate a new date format, and a sudden fall can be just as suspicious if the model has begun guessing where it previously abstained.

Decision monitoring follows the application outcome: reviewer corrections, rejected imports, duplicate records, and escalations. These signals are closer to usefulness, but they can be delayed or selectively observed — reviewing only rejected records gives a distorted view of accepted errors.

A small random audit of accepted records can uncover silent mistakes that rejected-record reviews miss, and it can supply future labels when the review policy and provenance are recorded. An “accept” click alone does not establish a clean training target, because acceptance and independent checking are different events.

Know when augmentation is the wrong repair

If OCR consistently drops minus signs or merges table columns, the extractor may never receive the necessary facts, and better training cannot recover an amount that disappeared from the input. Repairing the document-processing stage may be the correct intervention.

If reviewers disagree about which of two totals is authoritative, the policy is incomplete, and adding both interpretations to the training set teaches inconsistency. Resolve the business rule first, then relabel affected examples and evaluate the revised contract.

If the task requires currencies or payment terms the schema rejects, the output vocabulary is too narrow, and expanding it requires coordinated changes across the consumer, validator, labels, and model. An adapter cannot make a rejected enum value acceptable to the application.

If failures span many genuinely new document structures, more model capacity or a different document representation may help. The evidence should show that narrower fixes have been tried and that the missing capability persists. Scaling before diagnosing the input and policy can make the same mistake more expensive.

Exercises

1. Two totals, two policies

One invoice says “Earlier estimate USD 100.00; corrected final total USD 120.00.” Another says “Total USD 100.00; another total USD 120.00; neither is final.” Should both targets use null?

Answer

No. The first supplies a precedence rule and supports 120.00 if the task allows that wording. The second is unresolved and maps to null under this contract. Counting numbers is not enough; the relationship between them determines the label.

2. A better average

An augmented adapter improves overall exact match but invents more values where the gold is null. What should happen before promotion?

Answer

Inspect field and family results, then apply the application’s cost-sensitive gate. If unsupported values are especially costly, the candidate may fail despite a better average. Keep a separate measure for incorrect non-null values so an aggregate score cannot hide that regression.

3. Extra data or extra steps?

The clean arm uses 120 examples for two epochs. The augmented arm uses 184 for two epochs. Does the comparison isolate data diversity?

Answer

No. It changes coverage and optimizer steps. Add a matched-step control to study diversity more directly, while still reporting the fixed-epoch result as a practical training recipe. Neither design removes every difference in example exposure, so describe the question each one answers.

4. A green footer test

Every instruction-like footer in the challenge set is ignored correctly. Can the service be described as prompt-injection resistant?

Answer

No. The result covers those literal synthetic cases. Keep permission boundaries, validation, and review paths independent of the model. A broader security claim needs an appropriate threat model and evaluation, not a handful of successful extraction examples.

Final thoughts

A useful robustness test changes the information problem, not merely the spelling: it asks the model to preserve supported facts, ignore irrelevant text, and abstain when the source no longer determines an answer. The resulting error categories are a far better guide to the next dataset than a request for “more diverse examples.”

Next: Fine-Tuning for a Harder Generative Task