How to Evaluate RAG Retrieval Before Blaming the LLM

Your RAG system just produced a confident, polished, and completely wrong answer. The natural instinct is to blame the model—swap the prompt, change the temperature, upgrade to a bigger LLM. But before you touch the generator, ask yourself one question: did the retriever ever find the right evidence?

In a RAG pipeline, the model can only reason over the context it is handed. A wrong answer often means the retriever returned the wrong documents, not that the model hallucinated. Diagnose the retrieval layer first, because changing the prompt will not fix a retriever that never found the right material.

Why Wrong Answers Are Not Always the Model's Fault

RAG answers are downstream of retrieval. The generator does not search your knowledge base; it reads whatever context the retriever assembles and produces a response from that. If the evidence is missing, the model faces a choice: admit it cannot answer, or fill the gap with a plausible guess. LLMs are masters of the known, not pioneers of the unknown—give them the wrong context and they will confidently reason from it.

This creates two distinct failure classes:

• The right evidence was never retrieved. The answer-bearing chunk is sitting in your vector store, but the retriever returned something else.
• The right evidence was retrieved, but the model misused it. The context contains what the model needed, yet the generated answer still missed, contradicted, or ignored it.

The practical trap is that both failures look identical from the outside. You see a wrong answer, assume hallucination, and start tuning the prompt while the retriever keeps failing underneath. That is expensive, slow, and guaranteed to miss the real problem.

The decision rule: inspect what was retrieved before you change what generates. Retrieval evaluation comes first because it is cheaper to fix and because no prompt engineering can compensate for evidence that was never retrieved.

Two Criteria That Judge the Retrieved Context

When a RAG answer looks wrong, start by inspecting the retrieved chunks themselves. Two questions tell you whether the retriever did its job. A third question—which we will get to shortly—tells you whether the generator did its job.

Relevance: Is each retrieved chunk actually about the question being asked?

A chunk can be semantically similar to your query yet useless for answering it. Imagine asking "What is the refund policy for annual subscriptions?" and retrieving a chunk about how refunds work for monthly plans. The topic is adjacent—both are about refunds—but the specific fact you need is missing. Semantic similarity is not the same as usefulness.

Coverage: Is the complete set of facts needed to answer present across the retrieved chunks?

Some questions need evidence from multiple documents or multiple sections of one document. If the retriever returns one relevant chunk but the answer requires facts from three, the model is working with an incomplete desk. It will answer what it can and quietly fill the gaps.

Relevance and coverage judge the retrieved context. They answer one question: did the retriever hand the model the right material? If either fails, you have found your bottleneck—no prompt change will compensate for evidence that never arrived.

The Boundary Test: Could the Answer Come from This Context?

Once the retrieved chunks look relevant and complete, you need a different kind of check. This one does not judge the retriever. It judges whether the generator had a fair chance.

Run the boundary test: hide the generated answer, read only the retrieved chunks, and ask whether a careful reader could produce the correct answer from them alone.

This test separates two very different situations:

• The context is sufficient, but the answer is wrong. The retriever did its job. The model failed to reason correctly over the evidence it was given. This is a generation failure—the place where prompt tuning, model selection, or reasoning improvements actually matter.
• The context is insufficient, and the answer is wrong. The model never received what it needed. No prompt will fix an answer that requires evidence the retriever failed to deliver.

The distinction matters because beginners often blur these two checks. Relevance and coverage ask whether the retrieval succeeded. The boundary test asks whether the answer could have been supported by what retrieval found. A relevant, complete context does not prove the generator used it well—it only proves the generator had a fair chance.

A Worked Trace: Seeing the Criteria in Action

Here is what these criteria look like on a single query. Suppose a user asks: "What happens to my unused vacation days when I switch from the annual plan to the monthly plan?"

The routing decision falls out of the last column. Chunks 1 and 2 point to retrieval problems. Chunk 3 points to generation—or, as we will see next, to the source material itself.

Inspecting Retrieval Output Directly

Before building any metric pipeline, do the lowest-effort, highest-signal diagnostic available: bypass the generated answer entirely and look at what the retriever actually returned.

Run a handful of representative questions through your system and print the raw retrieved chunks—not the final answer. For each chunk, ask: does this actually help answer the question, or is it merely topically adjacent?

You will often find the classic failure: the retriever returns chunks that are on-topic but miss the answer-bearing sentence. The query was about a specific policy detail, and the retriever returned a general overview of the policy. The chunks look relevant at a glance. They are not useful.

Start manual before automated. A small eyeball pass on ten to twenty questions will reveal more about your retrieval quality than a metric you do not yet trust. You are looking for patterns: Does the retriever consistently miss a particular type of question? Does it favor certain document sections? Does it return chunks that are too coarse or too fragmented to carry the answer?

This inspection also tells you whether your evaluation effort is worth automating. If retrieval is failing on obvious cases, fix those first. Metrics will only confirm what you already see.

When the Source Itself Is the Problem

The retriever can do its job perfectly and the generator can reason flawlessly—and the answer can still be wrong. That happens when the knowledge base itself lacks what the question needs.

Three source-level failures masquerade as retrieval or generation problems:

• The fact is absent. No chunk in your corpus contains the answer. The retriever returns the closest available material, which is relevant but not answer-bearing.
• The fact is stale. Your documents describe an old policy, and the question asks about the current one. Retrieval succeeded against a knowledge base that is out of date.
• The source is contradictory. Two documents disagree, and the retriever returns both. The model has to pick a side without knowing which source is authoritative.

Before you change your embedding model or rewrite your prompt, confirm the needed fact exists in your corpus at all. Search for it directly. If the answer is not there, you have a knowledge-base problem, not a retrieval or generation problem. Label it as such and fix the source.

Measuring Retrieval with Hit Rate and MRR

Once manual inspection shows a systematic retrieval problem, you need to measure it. Two standard retrieval metrics turn your observations into something trackable.

Hit rate is the fraction of test questions where at least one relevant chunk appears in the top-K retrieved results. It answers a binary question: did the right evidence make it into the context at all? Think of it as retrieval recall at K.

MRR (mean reciprocal rank) measures how high the first relevant chunk ranks. If the best chunk appears first, that question contributes 1.0. If it appears third, it contributes 0.33. MRR rewards retrievers that put the best evidence near the top.

The tell is in the gap between them. A high hit rate with a low MRR means the right evidence is present but buried below irrelevant chunks. The model has to sift through noise to find what it needs—and it often will not. When MRR is the weak spot, reranking is the natural next lever, not changing your embedding model.

These metrics require a labeled set of questions with known-good chunks. That means building a small evaluation dataset where each question has a documented answer source. If you have not assembled one yet, that is the prerequisite to solve first. A metric is only as trustworthy as the test cases it runs on.

One limitation to keep in mind: hit rate assumes a single "best chunk" can be identified for each question. Multi-hop questions—those requiring facts from several sources—may need multiple supporting chunks labeled as relevant. If your test set only marks one chunk per question, your hit rate will understate how often retrieval actually gathered the full evidence picture.

Coverage: When the Right Chunks Are Not Enough

Relevance can look excellent while the answer still fails. The retriever returns on-topic chunks, every chunk is about the right subject, and yet the model produces a partial or wrong answer. This is a coverage failure.

Some questions are multi-hop: they require integrating facts from multiple sources. "What was the company's revenue growth after they acquired the competitor in 2023?" needs one chunk about the acquisition and another about revenue figures. Retrieving only the acquisition announcement leaves the model without the financial data it needs.

Coverage failures surface the chunking decisions you made earlier in the pipeline. Chunks too coarse bury specific facts under general context. Chunks too fine scatter related facts across fragments that never get retrieved together. Both break coverage, and neither is fixed by changing the generator.

The query itself can also be the bottleneck. A vague or multi-part question may need decomposition before retrieval can succeed. "Compare the pricing and features of the enterprise and pro plans" is really two retrieval tasks. If the retriever treats it as one, it will return a blend of chunks that covers neither question fully.

The decision boundary: if relevance is high but coverage fails, the fix lives in retrieval scope, chunk granularity, or query handling—not in the model.

A Practical Decision Sequence for Your First Debugging Pass

When a RAG answer looks wrong, run this ordered workflow from cheapest to most involved.

Step 1: Reproduce the wrong answer and capture the exact retrieved chunks for that query.

Do not work from memory. Log the query, the retrieved chunks, and the generated answer together so you can see the full chain.

Step 2: Score the chunks against the two retrieval criteria.

Is each chunk relevant? Is the complete set of facts present? Use simple labels—relevant, adjacent, or irrelevant for the first question; complete, partial, or missing for the second.

Step 3: Run the boundary test before routing.

Hide the answer. Could a careful reader produce the correct response from the retrieved context alone? If yes, the generator had a fair chance. If no, the retriever or the source failed first.

Step 4: Route to the failing layer.

• If chunks are irrelevant, suspect the embedding model, chunking strategy, or top-K setting.
• If chunks are relevant but incomplete, suspect coverage, chunk granularity, or the need for query decomposition.
• If the needed fact is absent or contradictory in the corpus itself, label it a source-quality problem before touching retrieval or generation.
• If chunks are complete and the answer is still wrong, the generator is the bottleneck.

Step 5: Escalate to hit rate and MRR only after manual inspection points to a systematic retrieval problem.

Build a small labeled set of questions with known-good chunks, then measure whether your fixes actually move the metrics.

The common mistake is fixing the symptom before confirming which layer produced it. You will waste days tuning a prompt for a retrieval failure, or worse, rebuild your chunking pipeline when the model was the problem all along.

Run one debugging pass on your own RAG system using these criteria before changing anything. Capture the chunks, score them, and let the evidence point to the bottleneck. Once you have confirmed that retrieval is the weak link, the path forward becomes clear: improve retrieval quality, and you will have given the generator the one thing it cannot manufacture—the right evidence.