Retrieval and RAG: Giving a Model Knowledge

A model knows what was in its training data and nothing else — not your company’s return policy, not last week’s prices, not the document a user uploaded a minute ago. Tools are one way to close that gap, by letting the model call an API for live or structured data. Retrieval is the other, and it’s the right one for unstructured knowledge. You fetch the text relevant to the question and put it directly in the prompt, so the model answers from it. Retrieval-Augmented Generation — RAG — is that pattern, and this chapter builds one end to end, then shows where it fits into a graph. Run against langchain-core 1.5.1, langchain-text-splitters, and langchain-ollama embeddings.

Two phases: index once, retrieve per question

RAG confuses people until they see it’s two separate jobs that happen at different times.

Indexing happens once, ahead of time. You take your documents, split them into chunks, turn each chunk into a vector (a list of numbers capturing its meaning), and store the vectors. This is setup — you do it when documents change, not per request.

Retrieval-and-generation happens on every question. You embed the question into a vector, find the stored chunks whose vectors are closest to it, drop those chunks into the prompt, and let the model answer from them. The model never “learns” your data — it’s handed the relevant slice at question time and reasons over it, then forgets it. Five verbs total: split, embed, store (indexing) and retrieve, generate (per question). Everything below is one of those five.

Documents and splitting

A Document is text plus metadata:

from langchain_core.documents import Document
doc = Document(page_content="Returns are allowed within 30 days with a receipt.",
               metadata={"source": "returns-policy", "topic": "returns"})

That metadata isn’t decorative — we’ll filter on it shortly. Real documents are too long to embed or retrieve as one blob: you’d blow the context window, and retrieval would be imprecise, because a whole manual is “sort of” relevant to everything. So you split them into chunks. RecursiveCharacterTextSplitter is the default workhorse — it tries to split on paragraph, then sentence, then word boundaries, keeping chunks near a target size:

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(chunk_size=120, chunk_overlap=20)
chunks = splitter.split_documents(long_docs)     # -> list[Document]

Two knobs decide retrieval quality. chunk_size trades precision against context: smaller chunks retrieve a tighter, more relevant slice but risk cutting a thought in half; larger chunks keep ideas whole but dilute relevance, so a match drags in surrounding noise. chunk_overlap repeats a little text between adjacent chunks, so a sentence split across a boundary still appears intact in at least one. There’s no universal right answer — it depends on your documents, which is why this is a knob a practitioner tunes rather than sets once. Verified: a document run through the splitter above produced 9 chunks of ~113 characters each. When retrieval quality is poor, this is the first place to look, before you blame the model.

Embeddings

An embedding turns text into a vector positioned so that semantically similar text lands nearby. That’s what makes retrieval work on meaning rather than keywords: “how do loyalty points work?” retrieves a chunk about earning points even though it shares no words with the query.

from langchain_ollama import OllamaEmbeddings
embeddings = OllamaEmbeddings(model="nomic-embed-text")   # local, no key
embeddings.embed_query("loyalty points")                   # -> [0.021, -0.118, ...]

You embed with a dedicated embedding model, not a chat model — different models for different jobs. This series uses nomic-embed-text locally; in production you’d swap in a provider’s embeddings the same way you swap a chat model. The one rule you cannot break: embed your documents and your queries with the same model. Two different embedding models produce vectors in incompatible spaces, and “closest” becomes meaningless — a silent failure that returns plausible-looking garbage.

A vector store, without the dependencies

A vector store holds the chunk vectors and does the similarity search. For learning and for plenty of production cases, InMemoryVectorStore from langchain-core is all you need — no external service, no extra package:

from langchain_core.vectorstores import InMemoryVectorStore

store = InMemoryVectorStore.from_documents(chunks, embeddings)   # index once
store.similarity_search("how do loyalty points work?", k=1)       # -> [Document, ...]

A deliberate choice worth flagging: older tutorials reach for FAISS or Chroma via langchain-community, but langchain-community is being sunset — it warns on import and points you to standalone packages. InMemoryVectorStore sidesteps that entirely for anything that fits in memory. When you outgrow it, the migration is to a standalone package (langchain-chroma, langchain-postgres) with the same interface — you swap the store, not the pipeline.

Seeing the scores, and filtering by metadata

Two capabilities turn a toy retriever into a usable one. First, you can see how relevant each hit is. similarity_search_with_score returns (document, score) pairs, so you can threshold — drop anything below a relevance floor rather than always taking the top k:

store.similarity_search_with_score("refund window", k=1)
# [(Document('returns within 30 days'), 0.51)]   # score is a float you can threshold on

Verified: the call returns a float score (0.51 here) alongside the document. When retrieval sometimes drags in a weakly-related chunk, a score threshold is how you cut it.

Second, you can filter by metadata — restrict the search to a subset of documents. If a question is about returns, search only the returns docs, so an unrelated-but-similar chunk can’t win:

store.similarity_search("anything", k=2, filter=lambda d: d.metadata["topic"] == "loyalty")
# only chunks whose metadata topic is 'loyalty' are considered

Verified: the filter restricts results to matching metadata. This is how real RAG scopes retrieval — by document source, by date, by the current user’s permissions — and it’s often the difference between a system that answers from the right document and one that confidently answers from the wrong one.

The retriever is a Runnable

store.as_retriever() wraps the store as a retriever, and a retriever is a Runnable — it has invoke, so it composes with the pipe like everything else:

retriever = store.as_retriever(search_kwargs={"k": 4})
retriever.invoke("how do loyalty points work?")   # -> [Document(...), ...]

The search_kwargs is where k and a metadata filter live for the retriever form, so the tuning above carries straight into the chain. Verified against the local embeddings, the query pulled back the chunk about earning points — a semantic match, not a keyword hit. Because the retriever is a Runnable, the whole RAG pipeline is one LCEL expression.

The full RAG chain

Now assemble it. The trick is running two things at once: fetch context for the question and keep the question itself, then feed both to the prompt. That’s exactly what RunnableParallel is for:

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnableParallel, RunnablePassthrough

prompt = ChatPromptTemplate.from_template(
    "Answer using ONLY the context. If it's not in the context, say you don't know.\n\n"
    "Context:\n{context}\n\nQuestion: {question}"
)

def format_docs(docs):
    return "\n\n".join(d.page_content for d in docs)

rag = (
    RunnableParallel(
        context=retriever | format_docs,   # fetch + join the chunks
        question=RunnablePassthrough(),      # pass the question straight through
    )
    | prompt
    | model
    | StrOutputParser()
)

rag.invoke("How many days do I have to return a book?")
# 'You have 30 days to return a book, per the return policy.'

Read the pipe: the question enters RunnableParallel, which retrieves-and-formats the context on one branch while forwarding the question on the other, producing {context, question}; the prompt fills its two holes; the model answers; the parser returns a string. Verified end to end against the local model — the correct chunk retrieved, a grounded answer. Every piece is a Runnable, so this chain streams and batches for free, exactly as Chapter 3 promised.

Grounding, and its failure modes

The prompt does real work: “answer using ONLY the context” is what keeps the model from falling back on its training and inventing a policy. This is RAG’s core discipline — the model answers from the retrieved text and says it doesn’t know when the text doesn’t cover the question, rather than confabulating. It’s not bulletproof. A model can still stray, but the honest, dominant failure mode is retrieval failure: if the right chunk isn’t in the top-k, the model answers from thin context and no grounding instruction can save it. This is why the tuning above matters — chunk size, overlap, k, score thresholds, metadata filters — and why real systems log what was retrieved for every answer, so a bad reply can be traced to a bad retrieval. When you later want to measure this, the evaluation chapter’s heuristic evaluators (“did we retrieve the chunk that contains the answer?”) are how. Grounding quality is a model-and-retrieval property, not a framework guarantee — the same honesty that applies to structured output.

Retrieval, tools, and where this goes in a graph

Retrieval and tools both give a model knowledge it lacks, and the line is clean. Use retrieval for unstructured, mostly-static knowledge you can index ahead of time: policies, docs, manuals, a knowledge base. Use a tool for structured or live data and for actions: the status of a specific order, today’s price, placing a booking. “What books are like this one?” is a retrieval question; “where is order A17?” is a tool question. The capstone uses both, retrieving over the catalog for recommendations while calling a tool for order status.

One forward pointer, because it changes how you’ll deploy RAG. The chain above is a pipe, fine for a pure question-answering endpoint. But inside an agent, retrieval is usually a node in a graph — a step that fetches context before the model reasons — so it can sit alongside routing, memory, and tool use rather than standing alone. The retriever being a Runnable is exactly what lets it drop into a graph node unchanged, which is how the recommendations specialist works in the capstone.

Final thoughts

RAG is five verbs — split, embed, store, retrieve, generate — wired with the pipe you already know. The parts that decide whether it works are unglamorous: chunk size and overlap, the value of k, score thresholds, metadata filters, and a prompt that pins the model to the retrieved context. InMemoryVectorStore keeps it dependency-free until you genuinely need a database, and the retriever being a Runnable means the RAG chain is just LCEL — and drops into a graph node when the time comes. That completes the primitives: messages, models, prompts, composition, tools, and knowledge. From here the flow stops being a straight pipe and becomes a graph.

Next: the StateGraph from scratch — state, nodes, and edges, built up from an empty graph, and the mental model that makes everything after it click.