Synthetic Data Generation

Chapter 10 showed why a longer training file need not teach the router a new decision. A larger model can write new examples for us, but their value depends on what the smaller model learns from them. That learning has to be judged on unfamiliar requests, not on the supply of examples.

The bookshop also needs to read supplier invoices. An extractor takes document text and returns named fields, such as the vendor and amount. We use that task here because a known invoice record gives us an answer key before generation begins. Chapter 13 develops the full extraction experiment; the small example below supplies the task knowledge needed for this data study.

The larger model is the teacher: it writes invoice text from the known fields. The smaller model is the student, which learns to recover those fields from the text using the accompanying answer. A fluent teacher response remains a candidate until we check that its wording supports the answer we intend to teach.

Run against: Qwen3-4B at the revision in scaling-models.json as a local NF4 teacher; Qwen3-0.6B float32 LoRA students, Transformers 4.56.2, PEFT 0.17.1, bitsandbytes 0.50.2, PyTorch 2.8.0, Apple M1 Ultra/MPS, seed 42.

Not run. A human-only versus synthetic-only versus human-plus-synthetic experiment, independent human review, and a held-out real-document benchmark are outside this study.

The available seed records are AI-authored or programmatically rendered teaching data. The measured arms are rendered-only, model-generated-only, and their mixture. Calling the rendered arm “human data” would misstate its provenance.

Follow one invoice from text to answer

The first clean training row, extract-train-000, contains this invoice text:

Invoice INV-1000. Vendor: Train Paper 0. Total: USD 100.00. Terms: NET_30. Date: 2026-01-01.

Its answer is a JSON object — the named-field representation that Python reads as a dictionary — and the six fields stay the same across examples:

{
  "invoice": "INV-1000",
  "vendor": "Train Paper 0",
  "amount": "100.00",
  "currency": "USD",
  "terms": "NET_30",
  "invoice_date": "2026-01-01"
}

Every value is a string or JSON null, which Python reads as None — and NET_30 means payment in thirty days. When the document omits a fact or leaves it unresolved, the corresponding target is null; the extractor must preserve that uncertainty. An invoice without payment terms should therefore produce "terms": null, even if most training invoices specify thirty days.

The task contract states which fields and values the application accepts. This study uses USD, EUR, or GBP for currency and NET_15, NET_30, or DUE_ON_RECEIPT for terms. Amounts are decimal strings with two fractional digits, and dates must be explicit, calendar-valid dates written year-month-day, the ISO form 2026-01-01 uses. The generator may change how those facts are written while the intended answer remains tied to the parent record.

A program already rendered the source text above, so asking the teacher to write another version gives us a second presentation of the same facts. The experiment compares learning from the program’s text, the teacher’s text, or both; neither source is an independently human-labeled invoice collection.

Decide what the generator is allowed to change

A generator can vary wording, layout, difficulty, and noise while preserving the task’s answer. For routing, the request must still require the intended source; for extraction, the text must still support the supplied fields. A SQL request and its query must likewise describe the same operation, or the variation changes what the example teaches.

These constraints suggest different generation strategies. A router example starts from a policy decision and a situation that makes the source necessary. An extractor example can start from a structured record and render it as text, and a SQL example can start from an executable query family and vary the natural-language request.

The invoice renderer separates facts from presentation: its source already contains the invoice reference, vendor, amount, currency, terms, and date. The teacher expresses those values in document-like form, with the original record retained as the answer key.

Keeping the answer key fixed reduces one source of error, but the teacher can still change what its words mean. It might describe the vendor as the buyer or place an amount in a sentence saying it was canceled. A field-presence check would find the value without establishing its role — the distinction the next filters need us to keep.

A proposal becomes a training row only after the task’s checks accept it. When a wrong answer has serious consequences, that review must establish the intended meaning as well as the presence of expected strings.

Ask for a narrow transformation

For the first record, the teacher receives the following instruction followed by the compact JSON answer above — the stored prompt wording, wrapped here for reading:

Render this invoice record as a brief email. Include each value
exactly once and unchanged. Do not add any other dates, amounts,
companies or instructions. No JSON, markdown or introduction.
Keep under 65 words. Record:

The saved response for generated-000 is:

Invoice: INV-1000, Vendor: Train Paper 0, Amount: 100.00, Currency: USD, Terms: NET_30, Invoice Date: 2026-01-01.

The parent is the known source record; the candidate gets a new ID and text while keeping its parent’s target JSON. The student learns to recover that target from the new wording, which is how this generation step supplies supervised examples. The response resembles a labeled note more than an email — preserving fields does not establish success at the requested style.

The teacher uses the pinned Qwen3-4B checkpoint, NF4 double quantization, float32 compute, non-thinking mode, greedy decoding, and at most 120 new tokens. Greedy decoding makes repeated generation repeatable under the same runtime, but it does not create diversity — that comes from the source values and the five style instructions.

Sampling chooses tokens from the model’s probability distribution instead of always taking the most likely token. Trying several temperature settings changes how concentrated that distribution is and could vary the wording further. Acceptance rate and quality would need evaluation under a new generation plan.

Every raw candidate stores the complete prompt, because “generated with Qwen” does not identify the instruction that shaped its wording. Model revision, decoding settings, parent ID, style, and review status travel beside the text so the row can be traced.

Filter in stages and keep rejection reasons

The companion first checks that each known field value occurs exactly once in the generated text, then checks for normalized-text duplicates and for a full student training sequence that fits the 256-token budget. Accepted candidates retain their original target object.

These mechanical filters catch missing or repeated values, exact duplicate renderings, and examples too long for the training recipe. Passing them establishes those properties and nothing about the sentence’s meaning, which still needs separate review.

A candidate can fail more than one check, so the raw record retains every detected reason instead of only the first. That lets rejection analysis distinguish an overlong candidate from one that also repeats values.

Trimming text or substituting a missing value changes the data construction method, even when the result looks easier to accept. Such repairs can be useful, but they belong in a recorded transformation with its own checks rather than in the reported generation pass.

Rejected text helps decide what to change before generating again. Missing payment terms point toward the prompt or source representation, and overlong candidates toward a shorter specification; repeated candidates suggest that another batch under the same settings would buy little variation.

Two candidates fail the field-occurrence check. Both replace the literal GBP code with £ and render DUE_ON_RECEIPT as “due on receipt.” This is generated-008, saved in full:

Invoice INV-1008 from vendor Train Paper 8 amounts to £108.56, due on receipt, dated 2026-09-09.

The wording is plausible, but it violates this study’s requirement to preserve every supplied value unchanged. Rejecting it says nothing about whether it is a good invoice sentence — only that it is outside the declared rendering protocol. A normalization study could accept such variants under a revised labeling policy, but it would need to define how symbols and natural-language terms map to the target vocabulary.

A filter can pass a sentence with the wrong role

Consider a parent record whose vendor is North Paper and whose amount is USD 420.50. A faithful rendering might say “Vendor: North Paper. Total: USD 420.50.” A misleading rendering might say “Buyer: North Paper. Disputed amount: USD 420.50.” Both contain the important strings, but only the first supports the intended vendor and total without another rule.

The occurrence filter finds the symbols in both sentences although their relationships to the task differ. That is why a passing row needs a precise status rather than an unqualified verified label — the filter has not checked meaning.

The next review step can ask two concrete questions for each field: which phrase supports this value, and does any other phrase contradict it? A reviewer who cannot point to supporting text should not approve the target merely because it matches the parent record, since the generator may have failed to express the parent faithfully.

Supporting spans — the phrase offered as evidence for each field — can make review more efficient, and they are review evidence rather than additional instructions to the student. A span checker confirms that the cited text exists; the reviewer still decides whether it supports the field.

Negative examples need equal care: if a generation request deliberately removes payment terms, the target must change to null. Keeping the old complete target would teach the student to invent the missing term. The data builder should record which transformation changed which field so an audit can distinguish intended uncertainty from accidental omission.

A useful rejection policy also separates repairable formatting from ambiguous meaning. Extra introductory prose can sometimes be removed without changing facts, while a swapped vendor and buyer cannot be fixed safely by deleting a phrase unless the intended source meaning is re-established. Automated cleanup should not make semantic edits invisible.

In this small teacher study, we keep the acceptance rule simple and state its limitation. You can then follow a known record through the prompt, raw candidate, filter decision, accepted row, student run, and reserved prediction. That lets you inspect what taught the student, instead of trusting an opaque file labeled high-quality synthetic data.

Add semantic review where mechanical checks stop

Semantic review compares the candidate with the task policy and parent facts to decide whether the target is still justified. Fluency alone cannot settle that question: a polished sentence can change entity roles or leave a contradiction unresolved.

Review effort can follow the risk in a candidate: straightforward renderings with intact fields may receive sampled review, while ambiguous or instruction-like cases receive closer inspection under a recorded sampling policy.

A second model can assist with review, but agreement between two fallible models is not independent human confirmation. Shared training patterns or a shared misunderstanding of the policy can lead both to accept the same mistake.

The executed study marks accepted examples as machine-filtered with no independent human review. This is a limitation of the dataset, not a reason to relabel the process as human-assisted after an AI system inspects its own output.

For a release-quality corpus, keep reviewer identity or role, decision, correction, and policy version in the provenance system appropriate to the organization. A rejected example can remain useful as an error case without becoming a supervised target.

Inspect the saved candidates

You can examine the generation step without loading either model. From the companion directory, save this as inspect_candidates.py and run python3 inspect_candidates.py; it uses only Python’s standard library and reads the published files:

import json
from pathlib import Path

root = Path("runs/generation-teacher-v1")
candidates = [json.loads(line) for line in
              (root / "candidates.jsonl").read_text().splitlines()]
accepted = [json.loads(line) for line in
            (root / "accepted.jsonl").read_text().splitlines()]

for candidate in candidates:
    if candidate["id"] in {"generated-000", "generated-008"}:
        print(candidate["id"], candidate["accepted"],
              candidate["reasons"])
row = accepted[0]
print("parent:", row["provenance"]["parent_id"])
print("target invoice:", json.loads(row["target"])["invoice"])

generated-000 True []
generated-008 False ['field_occurrence']
parent: extract-train-000
target invoice: INV-1000

The first two lines distinguish a mechanically accepted candidate from a rejected one, and the final lines trace the accepted row back to its parent and answer key. Adding print(candidate["raw"]) inside the conditional prints the rejected wording quoted earlier beside its recorded reason.

Build a generation matrix before requesting volume

“Generate diverse examples” leaves us unable to say which decisions the new rows cover. A generation matrix instead names the dimensions that matter and selects combinations deliberately, making both coverage and omissions inspectable.

Dimension	Router example	Extractor example	SQL example
Difficulty	Single source versus mixed source	One entity versus competing entities	Filter versus join or window
Length	Short request versus context paragraph	Compact note versus longer invoice	Short request versus detailed report
Explicitness	Named source versus implied source	Labeled field versus prose	Named aggregate versus business wording
Ambiguity	Missing operand or source	Conflicting date or total	Undefined business term
Language quality	Typo or shorthand	Misspelled labels	Informal phrasing
Noise	Quoted irrelevant instruction	Footer or email thread	Irrelevant reporting detail
Edge behavior	Clarification required	Null required	Unsupported schema concept

Not every cell should become a random combination — a short, noisy, ambiguous, multilingual request may be so underspecified that its label cannot be defended. Complexity is useful only when the expected behavior remains clear.

A few examples in each important cell let reviewers inspect the policy before the collection grows. Disagreement at that stage calls for a clearer policy or generation specification — otherwise one unresolved decision becomes many inconsistent labels.

The companion’s executed teacher study uses a deliberately small subset of this matrix: five rendering styles over known clean invoice records. Those styles are a brief email, plain invoice lines, a key-value note, a compact paragraph, and a semicolon-separated record, and the study claims no coverage of the other dimensions above.

Keep the seed’s origin visible

A seed set is the small collection that anchors generation, such as independently reviewed customer cases in a production project. Our seeds are synthetic teaching records with fields known by construction — which limits what their comparison can establish.

A comparison between rendered and model-generated text therefore studies two construction methods over the same known facts; it does not measure whether synthetic examples can replace expert annotation on real documents.

The thirty teacher prompts use the first thirty clean training records. The test and challenge records are never supplied to the teacher as demonstrations or source records, which keeps their exact values out of the generation input although the construction families remain shared.

Each generated candidate points back to its parent record, preserving a relationship that matters when splitting and analyzing the data. Ten renderings of one invoice still describe one set of facts, and if siblings cross train and test, familiarity with that parent can inflate the apparent generalization.

For a larger project, split parent groups before generating variants, keeping documents from the same source event, customer thread, or query family together — exact-text deduplication cannot detect every dependency that matters.

Decide what counts as a correct extraction

A generated dataset is useful only if the student can recover the right fields from unfamiliar inputs, and two scores say whether it did. Valid means that its answer satisfies the JSON and field rules; exact means that the valid answer also matches every field in the known answer key. Consider these constructed changes to the first invoice’s answer:

Student answer	Valid?	Exact?	What the check establishes
All six fields match the answer key	Yes	Yes	The entire record is correct
Vendor is Wrong Company; other fields match	Yes	No	The shape is acceptable, but the vendor is wrong
Currency is AUD; other fields match	No	No	The currency is outside the allowed vocabulary

The evaluator scores fields only after the whole answer passes the contract, so an invalid record receives no field credit even if some strings look right. This strict rule represents an application that cannot consume a rejected record, which is a different thing from counting recoverable facts in the raw text by hand.

The clean test uses tidy invoice layouts, while the challenge set changes features such as missing terms or conflicting totals. Its answer fields change with those features, and whole-record exact counts show how many complete invoices the student handles. A perfect clean score says nothing about whether the model knows when to leave a field unknown.

Compare sources under an explicit training budget

The rendered-only arm uses the parent records corresponding to accepted teacher candidates, the generated-only arm uses those accepted renderings, and the mixed arm contains both forms. Matching parent IDs keep the underlying facts aligned across the comparison.

Each student starts from the same Qwen3-0.6B base with a fresh rank-eight query/value adapter. The learning rate, seed, batch size two, alpha sixteen, and decoder are fixed, and every arm takes sixty optimizer steps.

This is a fixed-step comparison, so the mixed arm has more distinct text rows and therefore fewer repeated exposures per row within the same update budget. That difference is part of the question: does presentation diversity help under a fixed amount of optimization?

It is not a fixed-epoch or equal-data-volume comparison; a larger study could include both to separate data coverage from repeated exposure. The present design is small enough to inspect each generated row and each wrong prediction.

The clean test contains twenty-four records and the challenge set thirty-two, both reused across the three source arms. They share construction families with training, so the comparison remains a narrow study of rendering choices rather than an independent estimate of real-document quality.

To reproduce the training, use the companion’s frozen training environment on the supported MPS setup. This run loads the pinned 4B teacher and the 0.6B student, where the saved-file route above needed neither, so choose a new output prefix:

.venv/bin/python generation_study.py --prefix local-generation

The script writes candidates and accepted rows under runs/local-generation-teacher-v1/, then trains the rendered, generated, and mixed arms in directories sharing that prefix. Accepted rows are in accepted.jsonl; the published data/extractor/generated.jsonl remains unchanged. Existing output directories cause the run to stop rather than overwrite evidence. The companion README gives the pinned environment and artifact prerequisites.

Before reading the results, compare the budgets: sixty steps at batch size two present 120 examples in each arm. The mixed pool is twice as large, so its rows receive fewer repetitions on average. Predict what an equal score would establish — a tie under this update budget, rather than proof that the two sources teach the same behavior everywhere.

The teacher proposed 30 renderings; 28 passed the mechanical filters. The accepted rows retain their parent records and machine-only review status.

Measure	Rendered	Generated	Mixed
Training rows	28	28	56
Steps	60	60	60
Clean exact	24/24	8/24	24/24
Challenge exact	13/32	5/32	13/32
Train s	17.57	17.46	17.46

Interpret the result without changing the labels

The rendered-only and mixed arms both get 24/24 clean records and 13/32 challenge records exactly right, while generated-only gets 8/24 clean and 5/32 challenge records. Its clean outputs all satisfy the JSON contract, yet invoice accuracy falls to 8/24 and vendor accuracy to 16/24, with the other four fields correct on every clean record. The regression is about recovering particular field values, not about producing valid JSON.

Adding the generated renderings produces no measured accuracy gain over rendered-only under sixty updates, and using them alone loses accuracy on both sets. These results do not show that teacher-generated data is generally harmful: this is one small, mechanically filtered corpus, one seed, and a test that shares the rendered layouts. They do show why acceptance rate cannot stand in for downstream quality.

One saved clean-test error makes the regression concrete: for extract-test-001, the generated-only student returns "invoice":"Issued 2026-02-02" instead of the invoice reference INV-1401, with its other five fields matching. The output is valid because the invoice field accepts a string — it fails exact match because the string names the issued date rather than the invoice.

That error gives the next inspection a purpose: compare how invoice references and dates are presented in the training text, since more natural wording may help transfer while extra prose can obscure field roles. The saved error identifies a distinction to investigate; it does not establish which wording caused the model’s choice. It also tells the next batch what to exercise: the decisions the current student still gets wrong. More rows are useful only when their labels remain reliable and their variation addresses those decisions.

Apply task-specific validation to routers and SQL

Router generation needs semantic policy checks, because a requested DATABASE label does not make a generated request require private records. “Explain how a sales database works” remains a general explanation even if the generator was asked for a database example.

Useful router review asks what source is necessary, whether operands are supplied, whether a document takes precedence, and whether clarification is needed. Counterfactual pairs can test the boundary: replace “from our ledger” with explicit numbers and see whether the intended destination changes.

SQL generation needs executable validation. A teacher can propose a request and query, but the query should be checked against an independent expected result on discriminating fixtures. A query that runs and returns a number is not automatically the right label.

Later, we build the SQL corpus from controlled families and check it with an independent Python oracle, a program that computes the expected result another way. That gives synthetic construction a stronger answer check than free-form teacher agreement, although the chosen families still define the space we test.

Across tasks, the validation method should match the failure: JSON syntax checks structure, source-field checks support extraction, policy review supports routing, and execution plus an oracle supports SQL semantics. A generic “quality score” from the generator cannot replace these different checks.

Record rights and provenance before scaling distribution

A dataset’s technical usefulness does not determine whether it can be reused or distributed. Keep the source of seed records, model identity, applicable model license, and any generation-service terms associated with the dataset version.

The local teacher’s Qwen3-4B model card identifies an Apache-2.0 license, which describes the model repository’s stated terms and not rights in arbitrary source documents or every generated output.

If a hosted generator is used instead, record the applicable service terms and the date they were checked. If seed examples come from customer documents, follow the organization’s permissions and retention rules before generation or training, because synthetic wording can still preserve sensitive source facts.

Provenance also supports deletion and correction: if a parent record must be removed or its label changes, its generated descendants can be found. Without parent IDs, a later cleanup may leave near-duplicates scattered through the corpus.

Treat provenance as data that travels with the row, not as a paragraph added to a README after training. The most useful time to record it is when the candidate is created, while the prompt and source relationship are still available.

Exercises

1. Inspect three candidate renderings

Use the first invoice’s answer key. Candidate A is the saved generated-000 text. Candidate B changes only Vendor: to Buyer:. Candidate C changes only 100.00 to 100.01. For each, decide whether every parent value still occurs exactly once and whether the text supports the intended answer. Point to the phrase that settles your decision. B and C are constructed variations, not additional teacher outputs.

Answer

A preserves the values and explicitly identifies Train Paper 0 as vendor and 100.00 as amount. B preserves every value but assigns the company the buyer role, so occurrence checking cannot certify its meaning. C fails occurrence checking because the parent amount 100.00 is absent; it also contradicts that amount. For a saved rejection, generated-008’s text earlier in the chapter shows the two substitutions the filter caught.

2. A mislabeled control arm

The seed examples were authored by an AI assistant and rendered by code. Is it accurate to call the seed-only experiment “human-only”?

Answer

No. Name the actual source: AI-authored or programmatically rendered. A human-only comparison requires independently human-created or human-labeled data under a documented review process. Relabeling provenance would invalidate the interpretation of the experiment.

3. More rows, the same steps

The mixed arm has twice as many text rows but receives the same sixty optimizer steps. What changes besides source diversity?

Answer

The average number of exposures per row falls. The result compares source mixes under a fixed optimization budget, not equal epochs or equal per-example exposure. Report that design and add controls if the causal distinction matters.

4. A thousand siblings

A generator creates one thousand paraphrases from ten parent requests. Why is random row-level splitting risky?

Answer

Variants of the same parent can land in both training and test, making the evaluation reward familiar facts or decisions. Split parent groups first and preserve parent IDs. Exact-text deduplication alone does not remove that dependency.

Final thoughts

Synthetic generation is most useful when it expands a known decision space without hiding how the examples were made. Keep the parent facts, the prompt, the rejected candidates, and the review status. Then judge the dataset by what a student learns on a suitable reserve — not by how quickly the teacher fills a file.

Next: Failure-Driven Fine-Tuning