Building the Multi-Model Specialist System

Three accurate models do not automatically make one accurate application — a router can send an invoice to the wrong specialist, or a valid extraction can contain the wrong vendor. Each component needs a defined job, and the application needs a way to decide whether their combined result completes the request.

The bookshop’s orchestration layer connects those jobs: routing selects a capability, extraction proposes a record, and the SQL specialist proposes a query. Application checks then decide whether the input is suitable, access is allowed, and the result needs review. A larger model could handle requests outside these narrow contracts, but that escalation would need its own data boundary and acceptance rule.

Run against: Python 3.13.3, Transformers 4.56.2, PEFT 0.17.1, PyTorch 2.8.0, bitsandbytes 0.50.2; local Qwen3-0.6B router, augmented extractor, and quantized SQL adapters in pipeline_demo.py on Apple M1 Ultra/MPS.

Not run. An external general-model fallback, web search, a calculator service, live database access, a network API, and concurrent production serving are outside this integration demo.

Unsupported or rejected requests receive a local review status. The ten curated cases reuse component fixtures and demonstrate control flow, not an independent end-to-end quality benchmark.

Begin with responsibilities, not a graph of models

The router’s five labels — DATABASE, DOCUMENT, WEB, CALCULATOR, and GENERAL_LLM — identify the source or capability a request needs. They do not establish permission, confidence, or the particular operation a model supports under that route — those are the application’s to settle.

For example, DOCUMENT includes both invoice extraction and summarizing an attached contract, while our extractor supports only six-field invoices. Likewise, the SQL specialist accepts requests against the two-table bookshop schema; it cannot discover arbitrary databases or decide which results the caller may see.

A capability registry is the application’s mapping from supported operations to their handlers and required inputs. In this small demo, conditional branches implement that mapping: document payloads can reach the invoice extractor, and database payloads can reach the SQL specialist. Other routes go to review, keeping the implemented capabilities narrower than the router’s vocabulary.

Carry task data separately from routing text

A request envelope groups the instruction with its payload and application metadata, so each can be checked before dispatch. In a service, an authenticated principal means the user or service identity established by login or credentials. The local fixture instead supplies a Boolean authorization decision — a stand-in that shows where a real access check belongs.

The anti-join request from Chapter 15 appears twice in the pipeline, once denied and once authorized. Reconstruct the denied case before reading its path through the system. Save this as inspect_envelope.py in the companion directory and run python3 inspect_envelope.py; it needs only the standard library:

import json
from pathlib import Path

rows = [json.loads(line) for line in
        Path("data/sql/test.jsonl").read_text().splitlines()]
row = next(r for r in rows if r["id"] == "sql-test-07-anti_join")
case = {
    "id": "pipe-sql-2",
    "request": "Use our private bookshop database. " + row["text"],
    "payload_kind": "database",
    "row": row,
    "authorized": False,
}
print(case["id"], case["payload_kind"], case["authorized"])
print("source:", case["row"]["id"])

pipe-sql-2 database False
source: sql-test-07-anti_join

The router sees request; the specialist receives the task row only if the route, payload kind, and authorization allow it. The row carries fixture labels for offline scoring, but those labels do not select the route or correct a prediction, and the full demo’s expected_route field serves scoring only.

Here payload_kind and authorized are application-supplied fixture metadata. A real server must establish the received attachment, available database, and authenticated identity before making its access decision; a client-provided authorized=true cannot grant permission. Separating these fields also lets request identities and policy versions be logged without giving document text the same retention policy.

Make the control path explicit

The demo loads the router LoRA adapter, augmented invoice extractor, and 0.6B quantized SQL adapter, warming each before processing cases. The smallest SQL candidate clears the study’s illustrative gate, which lets the integration demonstrate what happens when an eligible model still makes a known semantic mistake.

For pipe-sql-2, routing returns DATABASE and the payload marker matches, but authorization is false. The pipeline stops before invoking the SQL generator and returns review with reason unauthorized. The otherwise identical pipe-sql-3 is authorized: it reaches generation, and the restricted executor accepts its query, leaving a candidate that still needs an intent check.

The database branch in Figure 18.1 places the stopping points around generation. A failed payload or permission check stops before the model call; a rejected query stops after it. Reaching the final candidate means the checks passed — correctness against the user’s request remains a separate decision.

Figure 18.1. Read downward from the DATABASE route. Missing or mismatched input and denied authorization stop the path before SQL generation. Executor rejection stops it afterward. The final candidate still needs an intent check: the authorized anti-join case reaches this state with an incorrect answer.

Stage	Successful output	Failure path
Router	One allowed label	Review for invalid or unsupported route
Payload check	Compatible task input	Review for missing or mismatched input
Authorization	Application permits database task	Review without SQL generation
Extractor	Strict six-field object	Review for invalid or incomplete record
SQL specialist	Restricted query result	Review for rejected query
Final acceptance	Application-specific confirmation	Review or stronger-model escalation

The table also includes the document path, where a parsed record must contain all required values to become a candidate. Its final row names an application responsibility; the local demo stops at candidate or review and implements neither a final semantic confirmation nor an external escalation. Calling the result candidate tells its consumer more than a generic success=true, which could mean anything from a completed model call to a completed task.

Keep authorization outside the router

A correct DATABASE route identifies a request for private records; the application determines whether the caller can use them. The denied anti-join case makes this separation observable, because its saved specialist prediction is absent rather than generated and discarded afterward.

For a service, the access decision should establish the permitted database, tables, rows, and operations, backed by credentials that enforce those limits independently. The fixture’s SQLite authorizer instead restricts actions and table reads within its disposable database — it supplies no tenant or customer access policy.

Document access needs the same separation: DOCUMENT cannot grant access to every file in a workspace. Retrieval and attachment handling must establish the allowed sources before their text reaches either the extractor or a fallback model.

Treat validation as several separate decisions

The extractor’s parser checks syntax, keys, types, allowed terms and currencies, amount formatting, and calendar-valid dates. The dispatcher then sends any record containing null to review, adding an all-fields-required workflow rule on top of the model’s contract.

A null can therefore be a correct extraction — and still prevent automatic completion. The saved pipe-doc-2 case illustrates this: the source omits payment terms, the model correctly preserves that absence, and the application asks for review. A complete record such as pipe-doc-0 instead becomes a candidate carrying needs_source_check, because parsing cannot establish which entity or amount the document supports.

The SQL path similarly separates executor acceptance from intent correctness. The executor rejects disallowed operations, excessive rows, or excessive work, but a permitted SUM query may still answer the wrong business question. Offline labels let us diagnose that distinction in the fixture; a deployed workflow needs its own acceptance policy when those answers are unavailable.

Inspect the saved paths before loading models

The saved trace contains the route, specialist output, guard decision, and offline grade for each case. Read the two invoice paths and the two authorization variants together to see which stage changes their outcome. Save this as inspect_pipeline.py and run python3 inspect_pipeline.py from the companion directory:

import json
from pathlib import Path

rows = [json.loads(line) for line in
        Path("runs/pipeline-v1/predictions.jsonl").read_text().splitlines()]
by_id = {row["id"]: row for row in rows}
for case_id in ("pipe-doc-0", "pipe-doc-2", "pipe-sql-2", "pipe-sql-3"):
    row = by_id[case_id]
    prediction = row["specialist"]
    decision = row["decision"]
    remaining = decision.get("reason")
    if remaining is None:
        remaining = ("source_check" if decision.get("needs_source_check")
                     else "intent_check")
    correct = prediction["exact"] if prediction is not None else None
    print(case_id, row["route"], "generated:", prediction is not None)
    print(" ", decision["status"], remaining, "offline exact:", correct)

pipe-doc-0 DOCUMENT generated: True
  candidate source_check offline exact: True
pipe-doc-2 DOCUMENT generated: True
  review incomplete_extraction offline exact: True
pipe-sql-2 DATABASE generated: False
  review unauthorized offline exact: None
pipe-sql-3 DATABASE generated: True
  candidate intent_check offline exact: False

The first two cases distinguish model correctness from workflow completeness — the last two show permission stopping one request before generation. The authorized request produces Chapter 15’s faulty anti-join and still reaches candidate, demonstrating why executor acceptance leaves an intent check. None means no specialist prediction exists for the denied request — it is not a failed model answer.

Let uncertainty produce a useful outcome

A review status should carry a reason the next actor can use, because different failures require different remedies. Missing authorization calls for an access decision, an incomplete invoice may need another document, and malformed output may justify a bounded retry — sending each to a stronger model would leave some of those problems untouched. Grouping every failure into one fallback bucket hides those distinctions from monitoring. More malformed responses suggest model or prompt behavior, while more missing payloads suggest an integration problem; the recorded reasons give the investigation a starting point. They also distinguish deliberate refusal from failure to perform an authorized task.

An external fallback would need a policy for the context it receives, the data allowed to leave the environment, and its output contract. Its answer would still require validation — escalation should preserve the original task boundary rather than granting the next model unrestricted authority.

Avoid using confidence as a permission slip

The router emits a label, without a calibrated probability that the route is correct. Calibration means checking whether predictions assigned a probability are correct at approximately that frequency on suitable data. A fluent explanation or a high next-token probability does not supply that measurement.

Even a calibrated routing score would address only routing, leaving extraction correctness, SQL intent, and authorization to their own checks. Multiplying unrelated confidence values would combine quantities whose meanings have not been established, producing precision without a reliable interpretation.

At this stage, the observable checks are more useful: whether the route is valid, the payload is present, the record parses, required fields are supplied, and the executor accepts the query. The saved anti-join shows their remaining blind spot — a plausible wrong answer can pass them all.

Confidence-based escalation would need evaluation of coverage, the fraction accepted, and selective error, the error rate among that accepted subset. Rejecting nearly everything can reduce accepted errors while increasing cost and reducing usefulness, so inspect both measures and sample accepted candidates as well as rejected outputs.

Read the integration result at the right scale

The full demo contains four document cases, four database cases, one general explanation, and one web request. Two invoices have missing or conflicting information, and the denied and authorized anti-join variants exercise different stopping points. These cases give the control path both candidate and review outcomes:

The router selects the expected destination on 10/10 curated cases.

Case	Actual route	Decision	Remaining check or reason
doc-0	DOCUMENT	candidate	source check
doc-1	DOCUMENT	candidate	source check
doc-2	DOCUMENT	review	incomplete extraction
doc-3	DOCUMENT	review	incomplete extraction
sql-0	DATABASE	candidate	intent check
sql-1	DATABASE	candidate	intent check
sql-2	DATABASE	review	unauthorized
sql-3	DATABASE	candidate	intent check
general	GENERAL_LLM	review	unsupported or wrong route
web	WEB	review	unsupported or wrong route

The offline oracle marks pipe-sql-3 incorrect even though the guard returns a candidate. Structural acceptance does not establish task correctness — the guard and the oracle answer different questions.

The reused component fixtures make this an integration rehearsal, with known labels available for diagnosing each stage. A fresh end-to-end benchmark would instead sample complete user tasks, including unsupported requests, missing attachments, mixed-source questions, and access contexts, then score the final outcome. A perfect specialist cannot help a request that never reaches it — the path has to be measured whole.

To rerun the model pipeline, first create the router adapter at runs/local-lora/artifact using Chapter 8, the augmented extractor using Chapter 14, and the 0.6B SQL adapter using Chapter 15. Weights are excluded from the published repository, so the frozen evidence directories alone cannot supply them. The following command uses those local artifacts and a new output directory:

.venv/bin/python pipeline_demo.py --output runs/local-pipeline \
  --router-adapter runs/local-lora/artifact \
  --extractor-adapter runs/local-extractor-augmented/artifact \
  --sql-adapter runs/local-sql-0.6B/artifact

The output directory must be absent; choose another local name for a repeat and adjust adapter paths if your earlier runs used different names. The script prints a JSON summary and writes predictions.jsonl plus results.json. The recorded summary contains these fields, shown as an excerpt rather than the entire model and path metadata:

{
  "n": 10,
  "correct_routes": 10,
  "candidate_count": 5,
  "review_count": 5
}

A fresh run supplies its own counts. To inspect it with the earlier activity, change its input to runs/local-pipeline/predictions.jsonl, keeping the frozen trace available for comparison.

Compose quality without assuming independence

Consider an illustrative batch of 1,000 document requests whose successful path requires both correct routing and correct extraction. If the router sends 950 to the right specialist, and extraction succeeds on ninety percent of those 950, then 855 complete both stages: 950 × 0.90 = 855, or 85.5 percent of the original batch.

The second percentage is conditional — it describes the requests that passed the first stage. No independence assumption is needed when that is the population measured. A standalone extractor score of ninety percent may describe easier or differently distributed documents, so it cannot automatically be substituted for success on the routed subset.

Routing and extraction difficulty may share a cause, such as ambiguous requests arriving with difficult documents. Measure the assembled path and retain intermediate labels for route correctness, specialist correctness, guard acceptance, and final task completion. Those labels explain where a request failed without replacing the final outcome metric.

Fallback adds another population: the rejected requests sent onward after the specialist path stops. Their difficulty can differ from a general model’s benchmark, so measure fallback success on that subset when estimating final quality. After using an integration suite to improve the system, keep it for regression checks and reserve fresh complete tasks for the release decision.

A mixed request needs a plan, not just a label

“Read this invoice, compare its total with our purchase order, and explain any difference” requires more than one specialist call. The router’s source-precedence policy selects the document first; orchestration must then represent the remaining dependencies rather than treating that label as a complete plan.

Extraction supplies the invoice reference and amount, authorized retrieval finds the purchase order, comparison checks the values, and explanation uses both sources. Each stage needs the earlier result: a missing reference blocks lookup, missing permission blocks access, and an unresolved amount blocks a meaningful numerical comparison.

The current demo handles one specialist task per case, leaving this multi-step workflow as an extension. Its intermediate states must preserve uncertainty: a null amount cannot become zero, and no matching purchase order cannot become a numerical mismatch. Each has a different remedy that the next stage needs to receive.

The plan also needs a stopping rule, because a failed lookup could otherwise trigger an unbounded sequence of generated queries. Cap attempts and preserve their reasons so clarification becomes a deliberate outcome. A larger model might help interpret the request or explain an established comparison, but it still depends on the source facts and permissions.

Extend the single-task path one dependency at a time, adding complete task cases as each stage becomes available. That keeps the evaluation attached to what the application can actually do, while the existing traces continue to test the original stopping points.

Measure the latency of the whole path

Routing precedes specialist inference, validation follows generation, and fallback may add a model call after the first attempt has consumed time. These sequential dependencies make component speed only part of the delay a user experiences — the assembled path supplies the relevant measurement.

The demo times a warmed path around routing, specialist work, and guards, with model loading outside the per-case timer. Its prediction helper also performs offline fixture grading, so this timing diagnoses the demonstration rather than isolating production serving latency. A service should separately record queueing, retrieval, external calls, and formatting to explain slow requests and fallback costs.

Keeping models resident reduces cold-start delay but increases memory pressure; loading adapters on demand can reverse that tradeoff. The demo uses three independent instances for clarity, while sharing a compatible base and switching adapters would require another serving design.

Such sharing also introduces a concurrency problem: one request could change the active adapter while another is still using it. That interference is a race condition, and preventing it requires isolation and tests under simultaneous requests. Lower memory use is useful only if each request still receives the intended specialist.

Make retries bounded and observable

A retry needs both a reason and a limit, since an identical greedy request often reproduces the same response. Changing a prompt or selecting another model is a new intervention whose effect should remain visible in the request trace.

For malformed JSON, a repair prompt may fix structure while inventing facts, and a postprocessor may accept a truncated record. Evaluate the final task correctness as well as parser acceptance. SQL error messages can likewise help a retry while exposing schema or infrastructure details, so the application’s data policy must govern what reaches an external service.

Repeated attempts must also preserve one request identity when a downstream action can change state. Idempotency means that retrying the same operation has the effect of performing it once; an invoice import should not create another invoice merely because its model call was retried. The fixture only reads data, while a real import needs this protection in application logic.

Record the original request, route, attempt number, model and adapter version, validation reason, escalation, and final outcome together. That chain lets the team locate the specialist failure behind an expensive retry instead of seeing only the last successful call.

Exercises

1. A document route without an invoice task

A user asks for a summary of an attached contract. The router chooses DOCUMENT. Should the invoice extractor run?

Answer

No. DOCUMENT is a broad source category, while the extractor implements a narrow invoice contract. The application must identify a supported operation or send the request to another capability or review path. A valid route does not imply that every specialist under that route is suitable.

2. A valid incomplete record

The extractor correctly emits null for missing payment terms. Is the model wrong when the dispatcher sends the record to review?

Answer

No. The model followed its contract, while the application’s all-fields-required workflow cannot complete automatically. Keep model correctness and workflow completeness as separate outcomes so review does not become a misleading model-error label.

3. A low fallback rate

A change reduces review requests from twenty percent to five percent. What else must be measured?

Answer

Measure errors among accepted candidates and final task completion. The change may improve capability, or it may simply weaken the guard. Coverage, accepted-output error, latency, and cost belong together.

4. A database label and a missing permission

The router correctly chooses DATABASE, but the caller lacks access. Should a larger model decide whether to proceed?

Answer

No. Authorization is an application decision. Stop before generation or execution as appropriate, and use the established access workflow. Model escalation cannot supply a missing permission.

Final thoughts

We can use small specialists without asking them to judge their own answers. Let them propose routes, records, and queries, while the application supplies authority, validation, and consequences. Keeping those boundaries visible is what makes the assembled system useful.

Next: The Economics of Small Models