The RAG Pipeline, End to End (and Run for Real)

The bookshop assistant has to answer “how long do I have to return a book?” correctly, every time, from your return policy — not from whatever a model absorbed about returns during training. The model has never seen your policy. It cannot have; the policy is private and it changes. So the architecture has to put the right policy text in front of the model at answer time, without dumping the entire company knowledge base into every request. That is Retrieval-Augmented Generation, and it is the center of Domain 3. This chapter teaches the whole pipeline and then runs it, so the numbers you see below are ones a real pipeline produced, not ones I’m describing.

What RAG is, and the alternative that doesn’t scale

RAG is a simple idea with a lot of engineering underneath. Before you ask the model a question, retrieve the handful of documents most relevant to it and include them in the prompt. Then instruct the model to answer from those documents. The model supplies language and reasoning; your corpus supplies the facts. The answer is grounded in text you control, which is exactly what you want when the facts are private, changing, or too voluminous to have been memorized reliably.

The obvious objection is: why not just paste the whole knowledge base into the context window? For a page or two, do — that’s the honest answer, and reaching for RAG when three paragraphs would fit is over-engineering. But the approach breaks the moment the corpus grows. A real support corpus is thousands of policies, catalog entries, and help articles. Stuffing all of it into every request is impossible past the context limit, and ruinously expensive per call where it fits at all. It is also worse at the task: a model handed a hundred pages to answer one question reasons less reliably than one handed the two relevant paragraphs. Retrieval is what keeps the context small, focused, and cheap. RAG scales; stuffing does not.

The whole pipeline is five stages: chunk the corpus into passages, embed each passage into a vector, index the vectors into a store, retrieve the passages nearest a query, and generate a grounded answer from them. The first three happen once, ahead of time; the last two happen on every question.

Chunking: cutting the corpus into passages

You don’t embed whole documents, you embed passages. A retrieval unit should be small enough that its vector represents one coherent idea and big enough that it stands on its own when handed to the model. So the first stage splits every document into chunks, and this unglamorous step decides more about retrieval quality than any other.

The two knobs are chunk size and overlap. Size is the tradeoff that matters. Chunk too large and each vector blurs several topics together, so retrieval gets imprecise: a query about refund timing pulls a chunk that’s half about shipping, and the model wades through irrelevance. Chunk too small and you slice a single fact across a boundary, so no one chunk holds the whole answer and retrieval brings back a fragment. Overlap repeats a little text between adjacent chunks. It hedges against that second failure: a sentence stranded at a boundary still appears whole in one of the two overlapping chunks.

Here’s the platform’s corpus running through a recursive character splitter, which prefers to break on paragraph and sentence boundaries rather than mid-word. This is real output:

from langchain_text_splitters import RecursiveCharacterTextSplitter

# CORPUS: 10 documents across three types (policy / catalog / help)
splitter = RecursiveCharacterTextSplitter(chunk_size=200, chunk_overlap=40)
chunks, metas = [], []
for doc_type, doc_id, text in CORPUS:
    for c in splitter.split_text(text):
        chunks.append(c)
        metas.append({"doc_type": doc_type, "doc_id": doc_id})

corpus: 10 documents -> 17 chunks (chunk_size=200, overlap=40)
chunks by doc_type: {'policy': 8, 'catalog': 4, 'help': 5}
  chunk_size=100 -> 32 chunks
  chunk_size=200 -> 17 chunks
  chunk_size=500 -> 10 chunks

Notice the chunk-size sweep on the same corpus: at 100 characters it fragments into 32 pieces, at 500 it collapses to 10 (barely more than one per document). The 200-character setting is a middle ground for these short policies; a corpus of long-form articles would want larger chunks. There’s no universal number — chunk size is tuned to your documents’ shape, and it’s the first thing to revisit when retrieval underperforms.

Notice too the metadata attached to every chunk: its doc_type and doc_id. That tag rides along through the whole pipeline and becomes the lever for the retrieval strategies in the next chapter. Attach metadata at chunk time or you won’t have it when you need it.

Embeddings: turning text into vectors

An embedding is a fixed-length vector of numbers that captures a passage’s meaning, produced by an embedding model. The property that makes retrieval work is that passages with similar meaning land near each other in the vector space, regardless of shared words. “How do I get my money back?” and “refunds are issued to your original payment method” share almost no vocabulary, yet their vectors sit close because they’re about the same thing. That’s the leap over keyword search, which would miss the match entirely.

Two architect-level facts. First, you embed with the same model at index time and query time — vectors from different models aren’t comparable. Second, the embedding model is a swappable component. This pipeline runs a local nomic-embed-text model so it costs nothing to verify. A production system would call a hosted embedding service such as Voyage AI, and the pipeline shape is byte-for-byte identical. The choice of embedding model affects retrieval quality and cost, not the architecture.

Indexing: the vector store

Embedding every chunk gives you a pile of vectors; the vector store is what makes them searchable. It holds each chunk’s vector alongside its text and metadata, and answers one question fast: given this query vector, which stored vectors are nearest? At small scale that’s a brute-force scan. At production scale it’s an approximate-nearest-neighbor index — the technology behind stores like pgvector, Pinecone, or Weaviate — that trades a sliver of accuracy for speed across millions of vectors.

The verification uses an in-memory store, which is the same interface without the operational weight:

from langchain_ollama import OllamaEmbeddings
from langchain_core.vectorstores import InMemoryVectorStore

emb = OllamaEmbeddings(model="nomic-embed-text")   # swap for a hosted model in prod
store = InMemoryVectorStore(emb)
store.add_texts(chunks, metadatas=metas)
# -> indexed 17 chunks into InMemoryVectorStore (nomic-embed-text)

Indexing is the last of the three build-time stages. From here on, everything happens per query.

Retrieval: finding the right passages

Retrieval embeds the incoming query with the same model, then asks the store for the k nearest chunks. k is the number of passages you pull; here it’s 3. Those chunks — and only those — become the context for the answer. This is the step that keeps the prompt small: out of 17 chunks (thousands, in production), the model sees the 3 that matter.

query = "How long do I have to return a book, and when will I get my refund?"
hits = store.similarity_search(query, k=3)

query: How long do I have to return a book, and when will I get my refund?
  retrieved [policy/returns]:      Returns policy. Customers may return physical books within 30 da...
  retrieved [catalog/book-refund]: The Art of the Refund, by J. Okafor. A witty business memoir abo...
  retrieved [help/refund-delayed]: Why is my refund delayed? Refunds start only after we receive an...

That result is worth staring at, because it’s honest about a real hazard. The top hit is exactly right — the returns policy. But the second hit is a catalog listing for a book titled “The Art of the Refund” — a genuine distractor. It scored highly on pure semantic similarity because it’s densely about refunds and returns, even though it’s a product, not a policy. Naive top-k retrieval pulls it in. It didn’t break the answer here, but it’s precisely the kind of wrong chunk that does break answers, and fixing it is what the next chapter is about.

Grounded generation: answering from the retrieved text

The final stage assembles the retrieved chunks into context and asks the model to answer only from it. The system prompt is doing real safety work: it forbids the model from answering out of its own parameters and tells it to admit when the context doesn’t contain the answer. That instruction is what turns a general model into a system that speaks only from your corpus.

import anthropic

context = "\n\n".join(f"[{h.metadata['doc_type']}/{h.metadata['doc_id']}] {h.page_content}"
                     for h in hits)
client = anthropic.Anthropic()
r = client.messages.create(
    model="claude-haiku-4-5", max_tokens=200,
    system=("Answer ONLY from the provided context. Cite the source tag you used. "
            "If the answer is not in the context, say you don't know."),
    messages=[{"role": "user",
               "content": f"<context>\n{context}\n</context>\n\nQuestion: {query}"}],
)

And the actual grounded answer the pipeline produced against claude-haiku-4-5:

Return window: You have 30 days from delivery to return physical books for a full
refund. The books must be unused and in original condition. [policy/returns]

Refund timing: Refunds begin only after the company receives and inspects your
returned item. During peak sale periods, warehouse inspection can add a few extra
days to the usual processing time. [help/refund-delayed]

Note that digital titles and gift cards are non-refundable. [policy/returns]

Every fact in that answer traces to a retrieved chunk, and the model cited which one. It correctly drew the return window from the policy and the refund timing nuance from the help article — two different documents, stitched into one grounded answer. And it quietly ignored the catalog distractor sitting in its context. That last part is luck as much as skill; the discipline of not handing the model distractors in the first place is the subject of chapter 9.

The grounding instruction is not decoration, and it’s worth being explicit about what it buys. Without “answer only from the context, and say you don’t know otherwise,” the model falls back on its parameters when the retrieved chunks come up short. It will answer confidently about your return policy using some generic return policy it absorbed in training, which may be flatly wrong for your business. That’s the worst failure a support system can have: a fluent, plausible, unsourced answer that a customer trusts. The instruction converts a silent wrong answer into an honest “I don’t know,” which is a failure you can detect and route to a human. Grounding is what makes retrieval quality observable downstream, and it’s why the citation in the output matters as much as the answer.

Final thoughts

RAG grounds a model’s answers in a private corpus by retrieving the few passages a question needs and generating strictly from them. It’s the scalable alternative to stuffing everything into context, which breaks on size, cost, and reasoning quality alike. The pipeline is five stages: chunk (size and overlap tuned to your documents, the single biggest lever on quality), embed (same model at index and query time, swappable for a hosted service), index into a vector store, retrieve the top-k, and generate under a strict grounding instruction. The run above is the real thing, distractor and all — which is exactly why retrieval done naively isn’t enough.

Next: getting retrieval right — k-tuning, metadata filtering, hybrid search, and the case where a wrong chunk breaks the answer and a filter rescues it.