Human Review, Confidence, and Provenance

Domain 5 closes with the reliability practices that decide when you can trust an automated system: calibrating human review (Task 5.5) and preserving where information came from through synthesis (Task 5.6). These are the Structured Extraction and Multi-Agent Research scenarios at their most rigorous. They’re design and measurement patterns, verifiable by reasoning.

The aggregate-accuracy trap

The foundational insight of 5.5, and a frequent exam trap: an aggregate accuracy number hides per-segment failure. “97% accurate overall” sounds like a system ready to automate — and it can be masking that the model is 99% accurate on invoices and 70% accurate on handwritten receipts, or excellent on the total field and poor on tax_id. The average washes out the segments that are failing, and if you automate on the strength of the aggregate, you ship the failures.

So the discipline is to measure accuracy by document type and by field before automating, not just overall. You verify performance is consistent across every segment — because a 70%-accurate field or document type is a liability no aggregate can excuse. Only once each segment clears the bar do you reduce human review for it. An exam item that justifies automation with a single high aggregate number is offering the trap; the right answer stratifies.

Sampling and calibrated confidence

Two techniques make review ongoing and targeted rather than exhaustive:

Stratified random sampling. To keep measuring error rates in high-confidence extractions — the ones you’ve stopped reviewing by hand — sample them randomly, in strata (by document type, by field), so you catch drift and novel error patterns before they spread. You’re not reviewing everything; you’re reviewing a representative slice continuously, which is how you notice when a previously-reliable segment starts failing.

Calibrated field-level confidence. Have the model output a confidence score per field, then calibrate those scores against a labeled validation set — so a reported “0.9” actually corresponds to ~90% real-world accuracy. Calibration is the crucial word: an uncalibrated confidence score is the unreliable proxy the escalation chapter warned against, useful only once you’ve checked it against ground truth. Calibrated, it becomes a routing signal — send low-confidence extractions and ambiguous or contradictory source documents to a human, and let high-confidence ones through — so limited reviewer capacity goes where the risk is. The exam distinction: raw model confidence is noise; calibrated confidence is a legitimate routing tool.

Provenance through synthesis

Task 5.6 is the reliability of multi-source work, and its core failure is subtle: source attribution is lost during summarization. When a synthesis agent compresses findings from several sources without preserving which claim came from which source, the report ends up with assertions no one can trace — you can’t tell whether “revenue grew 12%” came from the audited filing or a blog post. Once merged, the provenance is gone.

The fix is structured claim-source mappings that survive synthesis. Each subagent outputs its findings with their sources — URLs, document names, the relevant excerpt — and the synthesis agent is required to preserve and merge those mappings rather than flatten them. The claim and its source travel together, all the way to the final report, so every assertion remains traceable.

Two specific hazards the blueprint calls out, both about not fabricating a false clean answer:

• Conflicting statistics from credible sources. When two reputable sources disagree — 12% vs. 15% growth — the wrong move is to arbitrarily pick one. The right move is to annotate the conflict with source attribution: report both figures with who said what, and let the coordinator (or the reader) decide how to reconcile. A synthesis that silently chooses one value manufactures a false certainty; one that surfaces the disagreement is honest. Upstream, the analysis agent should pass conflicting values forward, explicitly annotated, rather than resolving them prematurely.
• Temporal differences misread as contradictions. Two sources reporting different numbers may not conflict at all — they might be measuring different periods. Requiring publication or data-collection dates in structured outputs prevents a 2024 figure and a 2026 figure from being flagged as a contradiction when they’re simply from different years. Dates are what let synthesis tell a real disagreement from a temporal one.

A final synthesis skill: structure the report to distinguish well-established findings from contested ones, preserving each source’s original characterization and methodological context — and render different content types appropriately (financial data as tables, news as prose, technical findings as structured lists) rather than flattening everything into one uniform format. Honest synthesis reflects the shape and certainty of what the sources actually said.

Final thoughts

Trusting automation is a measurement discipline: an aggregate accuracy number hides per-segment failure, so you validate by document type and field, keep measuring high-confidence output with stratified sampling, and route review by calibrated field-level confidence — raw confidence is noise, calibrated confidence is a tool. And multi-source reliability is provenance: preserve claim-source mappings through synthesis, annotate conflicts with attribution instead of arbitrarily picking a value, and carry dates so temporal differences aren’t mistaken for contradictions. The unifying reliability principle across all of Domain 5: be honest about what you know and where it came from — don’t let a summary, an average, or a synthesis manufacture a certainty the underlying data doesn’t support.

That completes the five domains. Next, we put them together the way the exam does.

Next: the six scenarios, worked — applying every domain to the production contexts the exam builds its questions from.