Observability and Evaluation: Seeing Inside a Run

The capstone works, but when it doesn’t — the supervisor routes wrong, an agent loops, a tool returns garbage — you need to see inside. A multi-agent run is many model calls, tool calls, and handoffs deep, and staring at a final answer tells you nothing about the path that produced it. This chapter is how you make a run visible, measurable, and testable, and it splits cleanly into three questions with three different tools: what happened in this run (tracing), is the system getting better or worse (evaluation), and does the graph’s logic do the right thing (testing). One of those — testing with fakes — is run-verified here; tracing and evaluation use LangSmith, whose API surface is confirmed but which needs an account this series has no key for, so those are marked accordingly. Run against langsmith 0.10.10.

Tracing the whole graph

LangSmith is LangChain’s tracing and evaluation platform, and it’s the default answer because instrumentation is nearly free: set two environment variables and every LangChain and LangGraph run is captured automatically, with no code change.

export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY=ls-...

With those set, running your graph records a trace: a tree with the graph invocation at the root, each node under it, each model and tool call under those, and on every span the inputs, outputs, token counts, latency, and any error. For the capstone, one trace shows the whole story — supervisor decided “orders”, the orders agent called run_sql with this query, got these rows, composed that answer — which is exactly the visibility a multi-agent system otherwise lacks. When a customer says “the bot gave me the wrong order status,” the trace is how you find out whether the model wrote a bad query, the query returned bad rows, or the model misread good rows. For your own functions outside the LangChain call graph, the @traceable decorator adds them as spans:

from langsmith import traceable

@traceable
def process_refund(order_id, amount):
    ...   # now appears in the trace

Verification note: the langsmith package and its traceable/Client surface are confirmed importable at the pinned version, but this series has no LangSmith API key, so the traces themselves were not captured against a live project. The auto-instrumentation and @traceable behavior are documented and stable; treat the specifics of the LangSmith UI as unrun here.

Cost and latency, per node

You don’t strictly need LangSmith to track cost, because the data is already in the messages. Chapter 2 showed every AIMessage carries usage_metadata with input_tokens, output_tokens, and total_tokens — verified there against a real run. Because it’s on the message, and the message is in the state, you can sum tokens per node by walking the message list, or accumulate them into a state channel with a reducer. In a multi-agent system this answers “which agent is burning the budget?” — the supervisor’s routing calls are cheap, but a recommendations agent reasoning over long retrieved context is where the tokens go. Tracing turns this into a dashboard; the raw numbers are yours for free either way, which matters because cost is one of the first things that surprises a team moving an agent to production.

Evaluation: measuring quality

Tracing shows one run. Evaluation measures many, against expectations, so you can tell whether a prompt change or a model swap made the system better or worse — the difference between “it seemed fine when I tried it” and a number you can watch. The LangSmith shape is a dataset of examples plus one or more evaluators:

from langsmith import Client
client = Client()

client.evaluate(
    my_agent,                       # the system under test
    data="bookshop-support-eval",   # a named dataset of {input, expected} cases
    evaluators=[correctness, no_hallucinated_orders],
)

Evaluators come in two flavors, and a real suite uses both. Heuristic evaluators are plain functions — did the SQL path return the right count, did the refund path refuse without approval, is the answer non-empty — cheap, deterministic, and right for anything with a checkable answer. LLM-as-judge evaluators use a model to score fuzzy qualities — is the recommendation relevant, is the tone appropriate — where there’s no exact match to test. The discipline that matters: score the consequential, checkable behaviors with heuristics (the refund gate must never be skipped is a heuristic you can assert, not a judgment call) and reserve the judge for genuinely subjective quality, because a judge is itself a model carrying the same reliability caveats as the system it’s grading. Grading a model with a model is useful but not ground truth. (As with tracing, evaluate is described here from its confirmed API, not run against a live project.)

Testing without a live model

Here’s the part you can — and should — run in ordinary CI, with no API key and no cost, and it’s verified. A graph’s control flow is deterministic even though the model isn’t: given a particular model output, the graph routes a particular way. So you test the graph by swapping the real model for a fake that returns scripted responses, and assert on the path. The simplest case asserts a reply:

from langchain_core.language_models.fake_chat_models import GenericFakeChatModel
from langchain_core.messages import AIMessage

fake = GenericFakeChatModel(messages=iter([AIMessage("scripted reply")]))
# build the graph with `fake` in the model slot, then:
assert app.invoke({"messages": [("user", "anything")]})["messages"][-1].content == "scripted reply"

The powerful case scripts a tool call, so you can test the whole ReAct loop deterministically — does the graph run the tool and loop back when the model asks? Script the fake to first request a tool, then answer:

scripted = iter([
    AIMessage(content="", tool_calls=[{"name": "lookup", "args": {"x": "A17"}, "id": "1", "type": "tool_call"}]),
    AIMessage(content="Here is your answer."),
])
fake = GenericFakeChatModel(messages=scripted)
# ...build the model↔tools graph with `fake`, then:
out = app.invoke({"messages": [("user", "status of A17?")]})
assert [m.type for m in out["messages"]] == ["human", "ai", "tool", "ai"]   # the loop ran

Verified: this passes, deterministically, no live model — the message types come back ['human', 'ai', 'tool', 'ai'], proving the graph saw the tool request, ran the tool, and looped back for the final answer. The technique generalizes to the whole system: script a fake to route a certain way and assert the supervisor hands off correctly; script the refund case and assert the graph pauses at the interrupt. This is what makes a multi-agent system testable at all — you separate “does the graph do the right thing given a model output” (deterministic, unit-testable with fakes) from “does the model produce good outputs” (non-deterministic, measured with evaluation). Test the first in CI on every commit; measure the second with datasets on a schedule. The model-agnostic slot that let this series verify against a local model is the same slot that lets your tests run against a fake — the property is the same, used two ways.

Debugging locally

Before any of the above, the fastest loop for “what is this graph doing” is streaming in updates mode and printing each step, or get_state_history to inspect the checkpoints after a run went wrong. For a system with persistence, the state history is a local trace you already have — every checkpoint, in order, with the state at each — so you can find the exact step where the state went sideways and even resume from just before it. Reach for LangSmith when you need production traffic, cross-run comparison, or a trace to share; reach for stream and get_state_history when you’re at your desk figuring out why one run misbehaved.

Final thoughts

Observability for a multi-agent system has three layers, and they answer different questions. Tracing (LangSmith, auto-instrumented) answers “what happened in this run” and, with token metadata, “what did it cost.” Evaluation (datasets plus heuristic and judge evaluators) answers “is the system getting better or worse” across many runs. And testing with fakes — the one you run every commit — answers “does the graph’s control flow do the right thing,” deterministically and for free, by separating the graph’s logic from the model’s judgment. Instrument with tracing, gate quality with evaluation, and let fakes keep the graph itself honest in CI. One chapter left: getting a system like this off your laptop.

Next: durability and deployment — durable execution and retries, and the LangGraph server that runs graphs as a persistent service.