Docs, Lineage, and Shipping It

The last mile: turn your project into living documentation with a real lineage graph, then run it in production — environments, CI, and building only what changed.

You’ve built a tested, layered project. One job is left before it can survive other people: let them understand it without reading every model. dbt does this almost for free, because the documentation generates itself from metadata you already wrote — every description, every column, every test, every ref(). This post is about turning that metadata into a browsable site with a real lineage graph, extending that graph past the last model to the dashboards that read it, and controlling what shows up. Actually running the project on a schedule is the next post’s job.

Documentation that writes itself

Here’s the payoff for all those YAML files. Every description, every column, every test, every ref() is metadata dbt already knows. dbt docs generate compiles it into a documentation site:

$ dbt docs generate
Building catalog
Catalog written to target/catalog.json
$ dbt docs serve      # opens the site on localhost:8080

You get a browsable catalog of every model and source — its description, its columns, the tests attached, and the compiled SQL that actually runs. Because it’s generated from the project, it can’t drift out of date the way a wiki does: edit a model, regenerate, and the docs match. The descriptions you wrote back in the staging and marts YAML files are what show up here, which is the quiet argument for writing them as you go.

The site is really two JSON files rendered by a static viewer, and the split is worth understanding because both files come back later. manifest.json is the project itself — every model’s compiled SQL, every test, every description, and the full ref()/source() graph. dbt writes it on any command that parses the project, so a plain dbt run produces one; it’s the same manifest that state comparison diffs against in the next post’s Slim CI. catalog.json is the part only docs generate produces: the warehouse’s own view of the tables — column types, comments, and (on adapters that expose them) row counts and sizes. Building it means querying the information schema, which is the “Building catalog” line and the reason docs generate costs a round trip even when nothing was rebuilt. The viewer overlays the two: structure and descriptions from the manifest, types and stats from the catalog.

That also tells you how to generate docs where there’s no live database to introspect — a CI job validating a pull request, say. dbt docs generate --no-compile --empty-catalog skips both the recompile and the catalog query; you get the full graph and every description from the manifest, and the column-type cells simply come up blank.

The lineage graph

The best part of the docs is the lineage graph — an interactive DAG of your whole project, built from the ref() and source() calls. Our bookshop draws itself, and the full graph is richer than the three-model sketch we’ve been carrying:

raw_customers ─▶ stg_customers ─────────────────▶ customers
raw_orders    ─▶ stg_orders ─┬─▶ orders ─────────▶ (feeds customers)
                             ├─▶ orders_by_customer
                             └─▶ order_events
raw_payments  ─▶ stg_payments ─▶ int_order_payments ─▶ orders

Click any node and the graph re-centers on it, fading everything that isn’t connected. The little --select box at the bottom of the viewer speaks the exact selector syntax from the node-selection post: type +orders and the graph narrows to orders and everything upstream of it; stg_orders+ shows stg_orders and its four descendants; +customers+ shows the full ancestry-and-descendants of customers. The graph and the CLI are the same graph — the viewer is just a mouse-driven front end for the selectors you already run in dbt build --select.

This is worth more than it sounds. When someone asks “if we change raw_orders, what breaks?” the graph answers it — trace the arrows downstream and you land on stg_orders, then orders, orders_by_customer, order_events, and finally customers. When a number in customers.lifetime_value looks wrong, you trace upstream — through orders, into int_order_payments, back to stg_payments and raw_payments — to find where the money entered the pipeline. New teammates read the graph instead of interrogating you. And none of it was drawn by hand; it’s a side effect of writing dependencies as ref() from day one.

Reading a model page

Click through to orders and you get the anatomy of a single model. The header carries the description you wrote — “One row per order, enriched with its total payment amount.” Below it, a columns table: order_id, customer_id, amount, each with its type (pulled from the catalog), its description, and the tests attached — so a reader sees at a glance that order_id is unique and not_null and that customer_id has a relationships test back to customers. Two panels, “Depends On” and “Referenced By”, are the lineage for just this node: orders depends on stg_orders and int_order_payments, and is referenced by customers. A Details strip shows the materialization (table), the relation name, tags, and the group the model belongs to.

The Code panel is the part engineers actually live in, and it has two tabs that are easy to conflate:

  • Source is the model as you wrote it — Jinja and all: the {{ ref('int_order_payments') }}, the {% set %} blocks, the macro calls.
  • Compiled is that same model after dbt resolved the Jinja: every ref() swapped for the real relation name ("bookshop"."main"."int_order_payments"), every macro expanded to plain SQL. This is the query you could paste straight into a SQL console.

There’s a third form the docs site doesn’t show, and it’s worth knowing exists: the run SQL — the compiled query wrapped in the DDL that materializes it (create or replace table … as (…)). You won’t find it in the docs UI; it lives on disk under target/run/bookshop/models/marts/orders.sql, next to the compiled-only copy in target/compiled/…. When you’re debugging why a model built the way it did, that pair — compiled/ for the pure query, run/ for the query-plus-DDL — is where you look, and the docs site’s Compiled tab is the same content as the compiled/ file.

Source and exposure pages

Models aren’t the only nodes with a page. A source — a raw_orders declared in a sources: block — gets its own entry, listing its tables, their columns and descriptions, and, if you configured freshness, the freshness thresholds and the most recent dbt source freshness result. The Referenced By panel on a source is often the more useful direction: click raw_orders and you see every model that reads it, which is the answer to “if the upstream team changes this table, whose models do I need to warn?” Our bookshop happens to load its raw data from seeds rather than a live source, so raw_customers and friends show up as seed nodes — same graph, same Referenced-By panel — but in a real deployment those first-column boxes would be sources, and the source page is where freshness lives.

An exposure gets a page too, and it’s the mirror image of a source’s: instead of “what reads this?”, it answers “what does this read, and who owns it?” The weekly_revenue_dashboard page renders its description, its maturity, a clickable url straight to the live dashboard, the owner’s name and email, and a Depends On list — orders, customers — that ties it back into the graph. It’s the one node whose whole purpose is to point out of dbt, which is exactly why it’s worth a page.

Column-level lineage, and where it stops

The graph you get from open-source dbt docs is model-level: nodes are models and sources, edges are ref()/source() calls. It will tell you that customers depends on orders. It will not tell you that customers.lifetime_value is specifically derived from orders.amount and nothing else — that’s column-level lineage, and the static docs site doesn’t compute it. Tracing a single column through the graph is a manual read of the SQL, model by model.

Column-level lineage does exist, but as a feature of dbt Explorer (the hosted metadata app in dbt Cloud), not the file-based docs site. Explorer parses the compiled SQL of each model to infer which input columns feed which output columns, and draws the finer graph. It’s genuinely useful — “which downstream columns break if I drop stg_payments.amount?” — but know its limits, because they follow from how it works: it reads SQL, so select * and select from a {{ dbt_utils.star() }} macro obscure the per-column trail (dbt can’t attribute columns it never sees named), and heavy expressions (a case folding five inputs into one output, a hand-written window function) get traced coarsely or not at all. Column lineage is an inference over your SQL, not a fact dbt stores — the cleaner and more explicit your select lists, the better it reads.

Our own pipeline shows both sides of that. customers.lifetime_value is sum(orders.amount) — a clean, named-column aggregate, and Explorer traces it cleanly back through orders.amount, int_order_payments, stg_payments.amount, to raw_payments. But orders_by_customer builds its columns with the count_by_status macro, which expands to a pile of count(case when status = … then 1 end) expressions at compile time; the tracer sees the compiled SQL, so it can attribute those to stg_orders.status, but the shape of the derivation — one status value per output column — is exactly the “heavy expression” case where a coarse depends on status is all you get. In the open-source site you’re working at the model grain anyway, and for most “what breaks?” questions that’s the grain that matters.

Richer docs: doc blocks and warehouse comments

The description: fields in your YAML are plain strings, which is fine until a description grows long, wants Markdown, or has to say the same thing on ten columns. Doc blocks move the prose into its own file. Define one with {% docs %} in any .md file under your project, then reference it with {{ doc() }} from the YAML:

{% docs order_status %}
The order's current state in the fulfillment flow — one of
`placed`, `shipped`, `completed`, `returned`, `cancelled`. Set by the OMS; not editable downstream.
{% enddocs %}
models:
  - name: orders
    columns:
      - name: status
        description: '{{ doc("order_status") }}'

Write the definition of “order status” once; every column that points at that block — stg_orders.status, orders.status, order_events.status — renders the same rich description in the docs site, and editing it in one place updates all of them. It’s ref()’s idea — define a thing once, reference it everywhere — applied to prose. (The list of statuses in that block is exactly the accepted_values set the staging test enforces, which is the nice thing about keeping the definition in one place: the docs and the test can’t disagree.)

doc() takes an optional package argument, too — {{ doc("shared", "order_status") }} — so a block defined in an installed package (an internal shared package of company-wide definitions, say) can be referenced from any project that depends on it. That’s the packages chapter’s idea reaching the docs: standard definitions of “revenue” or “active customer”, written once in a shared package and rendered identically wherever they’re used.

One doc block has a reserved name. A block called __overview__ replaces the generic text on the docs site’s landing page, so the first thing a visitor sees is your orientation instead of dbt’s default blurb:

{% docs __overview__ %}
# Bookshop Analytics

The models behind the bookshop's finance and growth reporting. Start with
the [lineage graph](#!/overview) to see how raw seeds flow into `orders` and
`customers`. Questions about revenue numbers → `#analytics-alerts`.
{% enddocs %}

It’s Markdown, links and all, and it’s the cheapest high-leverage documentation you’ll write: the one paragraph that tells a newcomer where to start reading a graph they’ve never seen.

By default those descriptions live only in the dbt docs site. persist_docs pushes them down into the warehouse itself, as native table and column COMMENTs:

models:
  bookshop:
    +persist_docs:
      relation: true
      columns: true

Now the description you wrote is attached to the actual table, where a BI tool or a colleague running DESCRIBE sees it — the documentation follows the data, even for people who never open dbt.

Finally, dbt docs generate produces a site made of several files served together, which is awkward to host statically or hand to someone as an artifact. --static collapses it into a single self-contained HTML file:

$ dbt docs generate --static      # → target/static_index.html, one portable file

That one file drops onto S3, GitHub Pages, or an internal docs bucket, and opens with a double-click — no server, no dbt docs serve.

Controlling what the docs show

Not every model deserves a spot on the graph. int_order_payments is an implementation detail — a rollup that exists so orders stays readable — and putting it in front of a business reader is noise. The docs config hides it:

# in the model's config, or under models: in dbt_project.yml
models:
  bookshop:
    intermediate:
      +docs:
        show: false

show: false drops the model from the docs site and removes its node from the rendered lineage graph. It’s cosmetic only — the model still builds, still gets tested, and other models still ref() it exactly as before; it just stops cluttering the picture the docs present. This is the docs-site equivalent of the private access level from the governance post: access controls who’s allowed to ref() a model across groups; docs.show controls whether it’s drawn. They’re independent knobs.

The same config sets a node’s color in the graph:

models:
  bookshop:
    staging:
      +docs:
        node_color: "#3b82f6"    # a named CSS color like "gold" works too
    marts:
      +docs:
        node_color: "#0d9488"

Color your layers and the DAG stops being a wall of identical boxes: blue staging feeding teal marts reads as a shape, and a stray blue node wired straight into a dashboard jumps out as the layering mistake it probably is.

The docs also surface the groups and owners from the governance post. orders and customers sitting in the finance group — declared with an owner back in the contracts chapter — show that group and its owner email on each model’s page, so a reader who finds a wrong number knows whose door to knock on without asking around. Ownership you declared once for access control doubles as documentation for free.

Exposures: lineage past the last model

The graph so far stops at your last model. But orders doesn’t stop there — it feeds a weekly revenue dashboard, and customers feeds a churn model somebody trains on a schedule. dbt has no way to know that from ref() alone, so those consumers are invisible: the graph goes dark exactly where the stakes get high. An exposure is how you tell dbt about a downstream consumer, so the lineage runs all the way to it. Declare them in a YAML file under your models path:

version: 2

exposures:
  - name: weekly_revenue_dashboard
    label: "Weekly Revenue"
    type: dashboard          # dashboard | notebook | analysis | ml | application
    maturity: high           # high | medium | low
    url: https://bi.bookshop.example/dashboards/revenue
    description: >
      Revenue and order volume by week, reviewed every Monday.
      If this is stale or wrong, finance notices first.
    depends_on:
      - ref('orders')
      - ref('customers')
    owner:
      name: Ada Lovelace
      email: [email protected]
    tags: ['revenue', 'finance']
    meta:
      slack_channel: "#analytics-alerts"

  - name: churn_model
    type: ml
    maturity: medium
    depends_on:
      - ref('customers')
      - source('crm', 'support_tickets')
    owner: {name: Data Science, email: [email protected]}

A few fields earn their keep. type classifies the consumer — dashboard, notebook, analysis, ml, application — and the docs site uses it to icon the node, so a dashboard and an ML model don’t look alike. maturity (high/medium/low) signals how much to trust it: a high-maturity dashboard is one people make decisions on; a low one is somebody’s experiment. url turns the exposure’s docs page into a click-through to the actual dashboard. depends_on is the load-bearing field — a list of ref() and source() calls that wires the exposure into the graph — and note the churn model depends on both a dbt model and a raw source, which is exactly the kind of dependency a diagram-on-a-wiki always forgets. owner, tags, and meta ride along as metadata; meta is free-form, so the Slack channel above is yours to consume in a notifier or a script.

Exposures aren’t built — there’s nothing to materialize — but they’re first-class nodes in every graph operation. In the docs site, weekly_revenue_dashboard appears as a terminal node hanging off orders and customers, so “what reads this model?” finally has a visual answer. And they participate in node selection:

$ dbt ls --select +exposure:weekly_revenue_dashboard
raw_orders
raw_customers
raw_payments
stg_orders
stg_customers
stg_payments
int_order_payments
orders
customers
exposure:bookshop.weekly_revenue_dashboard

+exposure:name selects the exposure and its entire upstream — every model that dashboard transitively depends on. That’s the “what feeds this dashboard?” query as a command, and it composes: dbt build --select +exposure:weekly_revenue_dashboard rebuilds precisely the models behind that one dashboard and nothing else, which is how you satisfy a “refresh revenue before the Monday review” request without rebuilding the project.

The larger payoff is that exposures make change-detection aware of the world outside dbt. Because the dashboard is a downstream node, a change that reaches orders now shows up as reaching the dashboard too — so the state-comparison CI from the next post can flag “this PR affects the weekly revenue dashboard” instead of letting a silent break sail through. “What breaks if I change this?” finally includes the things that read dbt’s output, which is usually where the expensive surprises live. The mechanics of wiring that into a CI check belong to the deployment post — here the point is just that the graph, and therefore every question you can ask it, now extends past your last model.

Keeping the docs current

Generated docs are only trustworthy if they’re regenerated. A docs site built by hand three months ago is a wiki with extra steps. The fix is to make dbt docs generate --static part of the same automated loop that runs the models — every production run rebuilds the single-file site and drops it on the docs bucket, so the hosted docs are never older than the last deploy. That loop — the scheduled runs, the CI checks, the state comparison that makes exposures pay off — is the whole subject of the next post; for now, the thing to internalize is that docs are an output of the pipeline, not a chore you do separately.

Everything in this series was run against DuckDB on a laptop. dbt docs generate, docs serve, the lineage graph, doc blocks, persist_docs, exposures, and --static were all exercised here. Column-level lineage (a dbt Cloud / Explorer feature) and the hosted-docs-in-CI loop are described from dbt’s documented behavior — they need Cloud or a CI environment to stand up.

Final thoughts

The last mile of dbt is where the earlier discipline compounds. Those descriptions become a documentation site; those ref() calls become a lineage graph; those exposures push the graph out to the dashboards and models that actually consume your work; those governance groups tell a reader whose number it is. Documentation is not decoration here. It is the map that lets a project survive more contributors, more consumers, and more change — and because it’s generated from the code, it’s the one kind of map that can’t lie about the territory.

The next step is operational: taking the graph, the tests, the docs, and the artifacts and putting them into a deployment loop that does not rely on a human remembering the right command.

Next: Shipping dbt Without Crossing Your Fingers

Comments