Retrieval Strategies: Getting the Right Chunks, Not Just Some Chunks

The previous chapter built a working pipeline, and at the end it retrieved a distractor: a query about refunds pulled a book titled “The Art of the Refund” into the model’s context. It didn’t break that answer. It will break others. The generation step can only be as good as the chunks handed to it. Feed it the wrong passages and it grounds a confident answer in the wrong text, which is worse than no answer at all. So retrieval is not a solved box you wire up once. It’s the stage where RAG quality is actually decided, and this chapter is about getting it right. Every result below is run.

Why retrieval needs a strategy at all

Naive top-k — embed the query, grab the k nearest chunks — is the right starting point and often enough. You reach past it when the shape of your data or the shape of your queries fights it. A mixed corpus is the classic trigger. When policies, product listings, and help articles all live in one index, a query’s nearest neighbors by raw meaning may be the wrong kind of document. A catalog blurb about privacy law reads a lot like a privacy-policy question. Precise-lookup queries are another: a user asking for an exact SKU or an error code wants a keyword match that vector similarity can fumble. The strategy you choose should answer a concrete failure, not be adopted because it sounds thorough. Bolting on reranking and hybrid search before you’ve seen retrieval miss is the same over-engineering trap the pattern-choice chapter warns about.

The landscape of retrieval strategies

Four moves cover most of what an architect reaches for, roughly in order of cost:

• Similarity-k tuning. The cheapest knob. k too small and you miss the passage holding the answer; k too large and you flood the context with marginal chunks that dilute the good ones and cost tokens. There’s a real sweet spot, and it’s specific to your chunk size and corpus.
• Metadata filtering. Constrain retrieval to chunks whose metadata matches — doc_type == "policy", or a tenant id, or a date range — before similarity ranking. This is the highest-leverage move for a mixed corpus, because it removes whole categories of wrong-kind chunks from contention rather than hoping similarity sorts them out. It’s why chapter 8 attached doc_type to every chunk.
• Hybrid search. Combine vector similarity with classic keyword (lexical) search and merge the rankings. Vector search wins on meaning; keyword search wins on exact tokens — names, codes, SKUs. Hybrid gets both. (Presented here as landscape; not run in this series’ harness — flagged so you know which claims are executed and which are described.)
• Reranking. Retrieve a generous candidate set with cheap vector search. Then pass those candidates through a heavier cross-encoder model that scores each one’s relevance to the query directly, and keep the top few. It’s the accuracy-for-latency trade you add when filtering and k-tuning aren’t enough. (Concept here; not run.)

Underneath these sits a design stance worth naming.

Progressive discovery vs. monolithic context

There are two philosophies for getting knowledge in front of a model. The monolithic approach front-loads: retrieve a big batch (or the whole document) and hand it over, trusting the model to find what it needs. Progressive discovery does the opposite. It retrieves a small, precise set for the immediate question, and if the model needs more, it lets the model ask — a follow-up retrieval, a tool call to fetch a specific document by id. The progressive stance is almost always the better architecture, for the same reason RAG beats context-stuffing: a small focused context produces better reasoning, lower cost, and lower latency than a large one. Precise retrieval is progressive discovery at the chunk level, which is why sharpening it matters more than widening it.

Running the strategies: where naive retrieval fails

Here’s a query engineered to trip naive retrieval, and it does. A EU customer asks the bookshop platform: “What is your policy on erasing a customer’s personal data on request?” The catch is that this phrasing reads almost exactly like the blurb for a book in the catalog: “Data and Privacy: A GDPR Field Guide … the right to erasure … deletion requests under European privacy law.” Watch what plain similarity does:

from langchain_ollama import OllamaEmbeddings
from langchain_core.vectorstores import InMemoryVectorStore
from langchain_text_splitters import RecursiveCharacterTextSplitter

# CORPUS = the same 10 policy / catalog / help docs from chapter 8
splitter = RecursiveCharacterTextSplitter(chunk_size=200, chunk_overlap=40)
chunks, metas = [], []
for doc_type, doc_id, text in CORPUS:
    for c in splitter.split_text(text):
        chunks.append(c); metas.append({"doc_type": doc_type, "doc_id": doc_id})

emb = OllamaEmbeddings(model="nomic-embed-text")
store = InMemoryVectorStore(emb)
store.add_texts(chunks, metadatas=metas)

query = "What is your policy on erasing a customer's personal data on request?"
for h in store.similarity_search(query, k=3):
    print(f"[{h.metadata['doc_type']}/{h.metadata['doc_id']}]")

NAIVE similarity_search(k=3):
  [policy/privacy]        <- correct
  [catalog/book-gdpr]     <- WRONG: a product listing, not a policy
  [policy/privacy]

The GDPR field-guide book lands at rank two, ahead of the second half of the actual privacy policy. On a mixed corpus, semantic similarity cannot tell “a document about data erasure” from “the policy on data erasure” — they mean nearly the same thing. That is the wrong-kind-of-document failure, and it’s not hypothetical, it’s the run.

The k knob, and how chunking hides the answer

Before filtering, look at what shrinking k does. Drop to k=1 and take only the top hit:

top = store.similarity_search(query, k=1)   # -> [policy/privacy] (first chunk only)

k=1 (top hit only):
  [policy/privacy]   first chunk: "...encrypted at rest and in transit.
                      We retain order records..."

The top chunk is the right document — but the wrong part of it. Chunking split the privacy policy in two. The sentence a customer needs (“under GDPR, customers may request deletion … within 30 days”) landed in the second chunk, which k=1 never retrieves. Feed only that top chunk to the model and here is the real grounded answer:

grounded answer from NAIVE k=1:
  I don't know. The provided context does not contain information about the policy
  for erasing a customer's personal data on request. The policy section only
  mentions that data is encrypted and that order records are retained for 7 years
  to meet tax obligations, but it doesn't address data erasure requests.

This is the honest failure mode, and it’s instructive on two fronts at once. k was too small to reach the answer, and the answer was split across a chunk boundary in the first place — a chunking decision and a retrieval decision compounding. The model did the right thing by refusing to invent a policy, which is the grounding instruction earning its keep. But a refusal is still a failed answer to a question the corpus can answer.

The metadata filter that fixes both problems

Now scope retrieval to policy documents only, with a one-line predicate on the metadata every chunk carries, and widen k enough to catch both halves of the policy:

policy_only = store.similarity_search(
    query, k=3, filter=lambda d: d.metadata["doc_type"] == "policy")
for h in policy_only:
    print(f"[{h.metadata['doc_type']}/{h.metadata['doc_id']}]")

FILTERED to doc_type=='policy' (k=3):
  [policy/privacy]   first chunk (encryption, 7-year retention)
  [policy/privacy]   second chunk (GDPR deletion, 30-day window)  <- the answer
  [policy/returns]

The catalog book is gone — it was never eligible, because the filter removed every non-policy chunk before ranking. And with k=3 inside the policy set, both privacy chunks come back, so the deletion clause is finally in context. The grounded answer against claude-haiku-4-5:

grounded answer from FILTERED policy retrieval:
  According to the privacy policy, under GDPR, customers in the EU may request
  deletion of their account and personal data. We complete verified deletion
  requests within 30 days, excluding records retained to meet tax obligations
  (which are kept for 7 years).
  [Source: policy/privacy]

Same query, same corpus, same model — a correct, complete, cited answer instead of a refusal, because retrieval handed generation the right chunks. That’s the whole thesis of the chapter in one before-and-after: the model didn’t get smarter, the retrieval got targeted. The filter did two jobs at once. It evicted the wrong-kind distractor, and by spending the k budget entirely inside the relevant document, it pulled in the split-off half the naive search missed.

Tying it to the platform’s mixed corpus

Hybrid search earns its place on this same corpus, even though it isn’t run here. Picture a customer asking about a specific title — “do you have Same-Day: Inside the Shipping Wars in stock?” A pure vector search can drift toward the shipping policy, because the query is dense with shipping meaning. It misses the catalog entry whose value is the exact title string. Keyword search nails the title token; vector search nails the intent; merging their rankings gets both. That’s the shape of query — an exact name, code, or SKU buried in otherwise-semantic text — that tells you to reach past pure vectors. It’s landscape here rather than a run, and flagged as such, but the trigger is concrete: keyword-precise lookups over a semantic index.

The lesson generalizes straight into the platform’s design. Because policies, catalog, and help articles share one index, metadata filtering is not optional, it’s the backbone. A question that’s clearly a policy question should retrieve under a doc_type == "policy" filter; a “help me do X” question should scope to help articles; a product question to catalog. Often you know the type from the route or a fast classification step before retrieval even runs. Filtering also carries the security load the next chapter formalizes. A tenant filter is what stops one customer’s retrieval from ever surfacing another’s data, enforced at the retrieval layer rather than hoped for at generation. Layer k-tuning on top (sized to your chunks, so a split answer stays whole), keep hybrid search in reserve for exact-token lookups, and add reranking only when a measured miss demands it.

Final thoughts

Retrieval is where RAG succeeds or fails, and naive top-k is a starting point, not a destination. On a mixed corpus, plain similarity confuses a document about a topic with the authoritative document on it. And too small a k can miss an answer that chunking split across a boundary. We watched both happen for real. Metadata filtering is the highest-leverage fix, because it removes wrong-kind chunks from contention entirely. K-tuning, hybrid search, and reranking round out the toolkit, each earning its place by answering a failure you actually observed. And the guiding stance is progressive discovery: retrieve a small, precise, filtered set, and let the system ask for more only when it must.

Next: security and authorization for RAG and tools — enforcing tenancy at the retrieval layer, scoping credentials, and keeping PII out of the wrong context.