Fine-Tuning for a Harder Generative Task

“Which books have never sold?” sounds like a short request. Answering it requires a relationship between books and sales, a definition of absence, and a query that preserves books with no matching sale. A model can produce fluent SQL that reverses that meaning — without making a syntax error.

Our third specialist translates a request into one SQLite query, a program that reads tables and returns rows. SQLite is the database engine used by the companion; the model supplies query text, while ordinary Python code executes and checks it. The application still decides which credentials and customer records are permitted, so we need that boundary before executing the model’s proposed program.

The extraction studies compared generated fields with a known record. SQL needs a different check: several programs can return the same answer, and a program can run successfully while answering the wrong question. The evaluator must therefore compare returned values, ordering and duplicates on data that exposes plausible mistakes. The goal here is to follow that evaluation and its worked failures rather than learn every SQL family from its name.

Run against: Python 3.13.3 SQLite, Transformers 4.56.2, PEFT 0.17.1, bitsandbytes 0.50.2, PyTorch 2.8.0; pinned Qwen3 models with NF4 double quantization and float32 compute on Apple M1 Ultra/MPS.

Not run. Live databases, multi-tenant authorization, large schemas, schema retrieval, arbitrary SQL dialects, and production query workloads are outside this experiment.

The companion executes only restricted queries over two disposable in-memory fixtures. Its small schema and twelve known query families do not establish general text-to-SQL ability.

Give the model the same schema the evaluator uses

A fixture is a small database whose contents we control for testing, and this one has two tables. Think of each table as records with named columns: books contains an ID, title, category, and integer price. sales contains an ID, book ID, quantity, and region. Prices are integer cents; a sale’s book_id points to its catalog book, and quantity describes how many units that sale records.

Table	Columns	Role
books	id, title, category, price	Catalog and unit prices
sales	id, book_id, qty, region	Sales quantities by region

The model receives this schema, the description of the tables and columns, in its system policy on every call. It must return a single SELECT query, which reads rows, or a WITH query, which names an intermediate query before using it in the final read. Prose and Markdown are excluded, and only the supplied tables and columns are allowed. The model can therefore consult the layout instead of trying to recall a private database structure from training.

The schema tells the model where values live — the request determines how to use them. “Revenue” requires finding each sale’s book, multiplying price by qty, then summing those contributions; counting sale rows would ignore the units and their prices entirely. “Books with no sales” instead keeps catalog rows whose IDs have no matching sale — the absence check, or anti-join, behind the opening request.

The schema leaves many business concepts undefined: there is no tax, refund, discount, timestamp, or customer table, so a request involving those concepts cannot be answered faithfully from this fixture. A production translator needs a clarification or unsupported-request path instead of an incentive to invent plausible columns.

Our reserved test stays inside the declared task, which keeps the training comparison controlled; recognizing unsupported questions would need its own labeled cases and acceptance rule.

Build a ladder of query difficulty

Query families isolate different decisions. Projection chooses output columns, such as book IDs and titles, and filtering keeps rows that satisfy a condition. Aggregation combines rows into a total, whereas a join matches related records across tables. A window calculation keeps the rows and computes a value within a group, such as each book’s rank in its category. These meanings let us read the following coverage table before studying the worked revenue and absence checks.

The companion defines twelve families, each with six training variants, one validation variant, and two test variants, which gives seventy-two training requests, twelve validation requests, and twenty-four test requests. In the table, HAVING means filtering totals after grouping, and a CTE, or common table expression, names an intermediate query through WITH. These are coverage categories; the worked examples focus on relationships between books and sales.

Family	Required behavior	Typical mistake
Projection	Return IDs and titles in order	Extra or missing columns
Price filter	Compare integer cents	Wrong boundary or unit
Category count	Count matching books	Count categories instead
Regional units	Sum quantities	Count sale rows
Join	Return distinct sold titles	Duplicate titles
Revenue	Sum price times quantity	Sum prices only
Grouping	Units per category	Wrong grouping key
HAVING	Filter grouped totals	Apply row-level WHERE
CTE	Top sellers with tie rule	Wrong ranking order
Window	Row number within category	Collapse rows with GROUP BY
Anti-join	Books without sales	Return sold books
Average	Mean price in a category	Integer or filter mistake

The table maps coverage rather than prescribing a universal order of difficulty: wording, schema, data and prior model knowledge all matter. A familiar window query may be easier for a pretrained model than an ambiguously phrased count. The worked queries concentrate on matching a sale to its book and detecting the absence of a match, where we can inspect exactly what goes wrong.

Every split uses the same families, with test requests changing framing and some literals rather than introducing new relational compositions. Some reference SQL strings are identical across splits — the operation has no varying parameter. The test therefore asks whether familiar operations survive new wording and values; it does not hold out whole programs as a test of composing new analytics.

An adapter may succeed by learning a compact mapping from familiar phrasing to query templates, which can be useful within a narrow application without establishing arbitrary analytical synthesis. Keeping the family coverage visible tells the application which kinds of request the comparison actually supports.

Make the database expose plausible wrong answers

A weak fixture can make a wrong query look right: if every book has a sale, both a correct anti-join and a query that always returns no rows can pass. If every sale quantity equals one, counting rows and summing quantities agree, and if every book has the same price, several revenue mistakes disappear.

The companion therefore uses twelve books but assigns sales only to the first ten book IDs, so books eleven and twelve remain unsold. Sale quantities vary, categories repeat, and prices differ — choices that make several common errors observable.

It also creates two fixtures using different deterministic seeds, so prices, sale-to-book assignments, quantities, and regional associations change. A candidate must produce the correct result on both — agreement on one database is insufficient.

Two fixtures are still finite examples, as tests are: a wrong query can agree on both, especially if they share structural regularities. The honest metric is “execution-correct on both declared fixtures” — not “mathematically equivalent SQL.” More independent fixtures, deliberately adversarial values, nulls, ties, and empty groups would strengthen the evaluator; the current pair favors clarity and inspection over broad semantic coverage.

Give the answer key an independent implementation

Using a reference SQL query as the only oracle creates a shared-failure risk: the generated query and reference may contain the same mistaken interpretation. If both multiply the wrong values, execution agreement can reward the error.

The companion computes expected results in Python from the fixture’s underlying book and sale records. For revenue, it looks up each sale’s book price, multiplies by quantity, filters by region, and sums. For unsold books, it compares catalog IDs with the set of sold IDs; for window ranks, it sorts each category explicitly.

The dataset builder first checks every reference SQL query against this Python oracle on both fixtures, and the evaluator then compares model query results with the oracle. This gives the answer key two implementations with different code paths.

They are not independent human judgments: both implementations were created for the same synthetic task and can share a conceptual misunderstanding, so independence here means implementation diversity, not independent annotation. A domain owner would still need to confirm what “revenue” means in a real bookshop.

The oracle also makes empty-result mistakes visible during dataset construction, because a green query that returns no rows is not enough — its result must match the expected values. Several useful bugs in data systems survive syntax checks and disappear only when someone asks whether the output contains the intended records.

Compare values, multiplicity, and order deliberately

Aliases give tables or expressions temporary names, so changing an alias or whitespace can change SQL text without changing the returned answer. Equivalent joins and harmless parentheses create the same problem for exact-string scoring. The primary metric should judge the requested result; otherwise it rewards a preferred spelling instead of the task the model was asked to perform.

The companion compares returned rows as ordered lists when the request specifies order; otherwise it uses a counter, which records how many times each row occurs — a multiset. A plain set would discard repeated titles, hiding a join that duplicated an answer, so keeping those repetitions makes the evaluator sensitive to the same multiplicity the consumer receives.

Column position remains significant — (title, id) differs from (id, title) when the consumer expects the latter. Numeric representation follows Python’s SQLite results, with integer cents avoiding monetary floating-point arithmetic in the fixture. The average family naturally returns a numeric average, so the comparison follows the returned value rather than the SQL string that computed it.

Null equivalence, decimal tolerance, timestamp zones, collation, duplicate handling and nondeterministic functions all need rules in a broader evaluator. Collation determines how text is compared and ordered; without a shared rule, even matching titles can be judged inconsistently. These choices belong to the task contract before model errors are inspected, so they cannot be adjusted afterward merely to improve a score.

“Top three sellers” leaves a decision unresolved when several books have equal units. The fixture specifies units descending and book ID ascending, so the ID breaks ties at the selection boundary. Without that second key, different valid query plans could choose different books even though each sorted by units correctly.

Put execution behind a restricted boundary

Checking that a string begins with SELECT is not a security boundary: a read query can still consume excessive resources, call functions, inspect metadata, or retrieve data the caller should not see. The companion uses the prefix check only as an early filter — nothing more.

The SQLite connection also has an authorizer. It allows the action types needed for reads and a small function allowlist, and rejects table reads outside books and sales. It denies writes, attachment operations, and other unapproved actions. It runs while a statement is being compiled, before any row is read, and it is distinct from the model’s instructions. See the SQLite authorizer contract.

A progress handler interrupts work after an approximate budget of SQLite’s internal instructions, so a runaway query stops. A row cap rejects answers that exceed one hundred rows. The runner uses execute, not executescript, for candidate SQL, so a multi-statement string does not become a sequence of operations; the progress-handler documentation explains the interruption mechanism.

These limits serve a tiny, disposable fixture and are not a complete sandbox for a live database service. A production system also needs caller authorization, database permissions, isolation, deadlines, resource limits, and a policy for which data may leave the system.

A route label cannot supply those permissions: if the router chooses DATABASE, it has classified the requested capability, not proved that the user may query every table. Keeping that distinction outside the model prevents a routing mistake from becoming an access grant — the two must stay separate.

Work through the revenue query’s grain

The request for total revenue in South combines catalog price with each sale’s quantity. Row grain means what one row represents: after this join, each row is a sale paired with its book. Multiplying price by quantity gives that sale’s contribution, and summing the contributions gives revenue under the teaching definition. The probe below checks the reference query against the independent expected result on each fixture, without calling a model. Save it as revenue_probe.py in the companion directory and run python3 revenue_probe.py:

from specialist_data import sql_cases, oracle
from specialist_eval import execute_sql

row = next(r for r in sql_cases("test", 2)
           if r["family"] == "revenue" and r["seed_parameter"] == 7)
print(row["text"])
for seed in (0, 1):
    actual = execute_sql(row["target"], seed)
    assert actual["ok"]
    assert actual["rows"] == oracle(row, seed)
    print(f"fixture {seed}: {actual['rows']} matches the oracle")

For the audit, answer this request: Calculate total sales revenue in cents for South.
fixture 0: [(31500,)] matches the oracle
fixture 1: [(33610,)] matches the oracle

The two fixtures give different revenues because their prices, quantities and regions differ, which is what lets them disagree with a wrong query.

The reference query is SELECT SUM(b.price*s.qty) FROM books b JOIN sales s ON s.book_id=b.id WHERE s.region='South'. Read it from the data relationship outward: the aliases name the book and sale tables, the join matches each sale to its book, and the region condition keeps South sales. Multiplication supplies each contribution and the sum combines them; matching on sale ID instead would pair unrelated records that happened to share an integer.

Summing b.price alone ignores the quantities sold, counting rows ignores both price and quantity, and summing the whole catalog includes books not sold in the requested region. Varying quantities and regional assignments makes these plausible alternatives return different answers, so a passing execution must do more than resemble the intended formula.

The fixture’s unique book ID prevents a sale from matching multiple catalog rows, whereas a real warehouse can contain duplicate dimension keys or historical versions, causing a join to multiply amounts. The model’s query might be syntactically correct while relying on a uniqueness assumption the actual database does not satisfy, so schema documentation should state keys and relevant grain, not merely column names.

The business definition is intentionally limited to current catalog price times quantity. A real revenue report might require the price recorded at sale time, discounts, refunds, and taxes, none of which exists here. That distinction is easy to lose when generated SQL looks sophisticated: a query can express the declared formula perfectly while the formula is unsuitable for the business question. Domain review therefore belongs before model evaluation.

Train a query completion under a fixed budget

The SQL training path reuses completion-only loss: the system policy and request are masked, while the reference query and end token receive loss. The model is learning to produce a program from the supplied task context, just as the extractor learned to produce a record. Every encoded sequence must fit the fixed 256-token budget, so no target query is silently cut in half.

The controlled SQL study uses one epoch, batch size one, and seventy-two optimizer steps for each model size. It returns to the QLoRA setup: an NF4-quantized base with double quantization and float32 computation, rather than the extractor’s unquantized base. Rank-eight adapters target query and value projections, with alpha sixteen and learning rate 0.0002; holding this recipe across sizes prepares the next comparison without claiming it is optimal for each model.

The small budget is intentional: it asks how much a short, consistent specialization pass changes each model, not what the best possible recipe for every size would achieve. A larger model may prefer another learning rate or adapter configuration, and a smaller one may need more optimization.

Before training, the runner evaluates the quantized base on the test; after training, it evaluates the final adapter on the same cases, with no test-selected checkpoint or hyperparameter search within this plan. Once these results are inspected, the set becomes part of development knowledge for future experiments.

.venv/bin/python specialist_run.py \
  --task sql --size 0.6B --quant --before --train \
  --epochs 1 --batch 1 --output runs/local-sql-0.6B

The --before flag records base predictions in the same run directory, alongside separate after-training predictions, losses, configuration and result files. The table can then be read as a comparison of the quantized base with its adapter under the same task. “Valid on both fixtures” means execution passed the restricted runner on each database; “execution-correct” additionally requires the returned answers to match the oracle.

0.6B arm	Execution-correct	Valid on both fixtures	Median inference ms
Quantized base	1/24	18/24	1422.7
QLoRA adapter	22/24	24/24	2063.2

Diagnose the smallest model’s errors

A query can fail before execution by including Markdown or naming a nonexistent column, or it can execute and use the wrong aggregate. Even the right values can fail an ordering requirement. Locating the first failing layer keeps an output-format repair from being mistaken for a fix to the query’s meaning.

These categories suggest different interventions: output-format failures can respond to examples or constrained generation, and schema mistakes may need clearer schema context or retrieval. Wrong joins and aggregation may require better examples, a stronger base, or a narrower supported query language.

A lower token loss can reflect better SQL punctuation while leaving a rare semantic distinction unresolved, so execution grading connects those training changes to the actual task by checking the returned answer. The loss curve remains useful for optimization — but it cannot identify which request the program has learned to answer.

Per-family results also prevent easy queries from hiding hard ones: if projection and simple filters are perfect but anti-joins fail, a single average cannot justify broad database access. The application could support the reliable subset while sending unsupported operations to another path.

The two remaining errors in the 0.6B adapter are the anti-join requests, and it emits the same query for both:

SELECT id FROM books
WHERE NOT EXISTS (
  SELECT 1 FROM sales WHERE sales.book_id = id
)
ORDER BY id

Within the subquery, the unqualified id resolves to sales.id because that column exists in the inner scope. The condition therefore compares values inside the sales table instead of asking whether a sale belongs to the current outer book. The intended correlation, that dependence on the book being tested, has disappeared. You can watch the consequence from the companion directory; the restricted executor runs the faulty query on both fixtures:

from specialist_eval import execute_sql

faulty = "SELECT id FROM books WHERE NOT EXISTS (SELECT 1 FROM sales WHERE sales.book_id = id) ORDER BY id"
for seed in (0, 1):
    result = execute_sql(faulty, seed)
    print(f"fixture {seed}: ok={result['ok']} rows={[r[0] for r in result['rows']]}")

fixture 0: ok=True rows=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
fixture 1: ok=True rows=[]

On fixture zero the query returns all twelve books; on fixture one it returns no rows, while the oracle expects book IDs eleven and twelve on both. The executor reports ok for both, because nothing about the query is illegal.

The reference restores the missing relationship with sales.book_id = b.id, where the outer table is books b. For each outer book, the inner query now checks for a sale of that book; NOT EXISTS keeps the book only when that check finds none. Qualifying the name changes which question the program asks — even though both versions are syntactically valid.

Figure 15.1. Read upward. The reference condition correlates each inner check with the outer book row, so NOT EXISTS keeps a book only when no sale names it. The faulty condition binds the unqualified id inside the sales table, compares two sales columns, and never looks at the outer row. Teal marks the correlation; red marks the reference that stays inside the inner scope.

The query passes the restricted executor and returns a legal result, so neither JSON formatting nor Markdown explains the failure — SQL name resolution and correlation are the missing distinction. More examples, a constrained query template or a stronger model are plausible interventions; the execution evaluator can compare them without treating this error alone as proof of a capacity limit.

The case also shows why an empty result deserves inspection. If a fixture had no unsold books and the inner condition happened to find a match, both the correct and faulty queries could return an empty result. Keeping two unsold books makes the mistake visible without requiring a large database.

Before comparing model sizes under the same recipe in the next chapter, we need the evaluator to show us how the small model fails. Otherwise we would be buying more capacity in response to a disappointing number, without knowing what capability we need it to add.

Use generation to propose SQL, then validate the proposal

A capable teacher can expand SQL training data, but its answer is still a candidate. The useful generation unit includes a natural-language request, schema, proposed query, and a way to determine the expected result. A plausible query alone is not a labeled example — it is a proposal awaiting a check.

Execution filtering removes syntax errors and some semantic errors, and it becomes stronger when the query is checked on several fixtures designed to distinguish common mistakes. A teacher that supplies both the query and its own claimed answer has not created an independent oracle.

For narrow tasks, controlled query families can be safer than unrestricted generation: start from a known operation, vary parameters and wording, and validate that the requested semantics remain unchanged. The companion uses this approach for the SQL corpus. The model’s role is translation within a declared family space.

The cost is limited breadth, because a family generator cannot reveal operations its author never included. Human review, independently sourced requests, and new compositions are needed before claiming wider capability. Synthetic scale should make a sound task definition easier to learn — not conceal the absence of one.

Exercises

1. The empty anti-join

Every book in a fixture has at least one sale. Why is that a weak test for “books with no sales”?

Answer

The correct answer is empty, so a query that always returns no rows can pass. Include unsold books and verify their expected IDs. A second fixture should change relationships so accidental agreement becomes harder.

2. A harmless alias

Two SQL queries differ only in aliases and whitespace but return the same ordered rows on both fixtures. Should exact-text mismatch make one incorrect?

Answer

No. The primary task metric compares results under the declared ordering rule. String comparison can remain a diagnostic, but it should not penalize equivalent spellings. Agreement on two fixtures still does not prove equivalence on every possible database.

3. Duplicate results

A join returns each title twice. A set comparison says it matches the gold titles. What is wrong with the evaluator?

Answer

A set discarded multiplicity. Use an ordered list when order matters or a multiset when it does not. Whether duplicates are allowed is part of the query contract, not a formatting preference.

4. A trusted route

The router emits DATABASE and the generated query begins with SELECT. Is it ready for a customer’s production database?

Answer

No. The route identifies a capability, and the prefix is only an early syntax filter. Caller authorization, allowed data, database permissions, resource limits, and result handling remain application responsibilities. The book’s fixture runner grants none of that production authority.

Final thoughts

With SQL, we can see why the evaluator matters so much: a fluent program may execute successfully and still answer the wrong question. To trust this specialist within its task, we need a clear schema, independent expected results, fixtures that distinguish plausible mistakes, and a boundary limiting what the generated program can do.

Next: When the Small Model Hits Its Limit