RAG Answer Quality Evaluation: Faithfulness, Relevance, and Completeness

Your retriever finally returns the right documents. The chunks look relevant. The context window is full of plausible evidence. And the model still hands you an answer that is wrong, off-target, or only half of what you asked for.

It is tempting to blame the model. But the real problem is often closer to home: you are evaluating the wrong contract. Retrieval quality and answer quality are two different jobs, and you cannot debug one by staring at the other. This article shows you how to evaluate grounded answers using three distinct lenses—faithfulness, relevance, and completeness—so you can name the actual failure before you change anything.

Why a Good Retriever Can Still Produce a Bad Answer

Here is the situation that brings most people to RAG answer evaluation: retrieval returns plausible chunks, the context looks right, and the generated answer still misses the mark. The answer might invent a detail that is not in the evidence. It might answer a different question than the one you asked. Or it might cover one part of a multi-part question and quietly ignore the rest.

The misconception is that a good retriever should guarantee a good answer. If the right documents were found, the reasoning goes, the model should be able to produce the right response. So when the answer disappoints, the model gets the blame.

That instinct is wrong in a useful way. Retrieval and generation are separate contracts with separate failure modes. Retrieval evaluation—context precision, context recall, context relevance—tells you whether the right evidence was found. It says nothing about what the generator did with that evidence once it arrived. A fluent wrong answer is not a retrieval failure. It is an answer-quality failure.

Think of it this way: retrieval decides what evidence is on the desk. The generator decides what it does with that evidence. You evaluate each contract separately, because each one fails for different reasons and gets fixed in different places.

When an answer disappoints, run it through three answer-side lenses before you touch the retriever or swap the model:

• Faithfulness: Is every claim in the answer supported by the retrieved context?
• Relevance: Does the answer actually address the question that was asked?
• Completeness: Did the answer cover everything the question required?

Each lens catches a different failure. Each one points to a different fix.

Faithfulness: Is Every Claim Supported by the Retrieved Context?

Faithfulness measures whether the answer's claims are grounded in the retrieved evidence. No invented facts. No contradictions. Every statement in the answer should trace back to something in the context you provided.

The practical way to check faithfulness is to break the answer into individual claims, then check each claim against the retrieved chunks. A claim that appears in neither the context nor the question is a faithfulness failure. If the context says a product has three pricing tiers and the answer says four, that is a faithfulness failure. If the answer adds a detail about refund policy that appears nowhere in the retrieved documents, that is also a faithfulness failure.

Here is the part that surprises people: an answer can be perfectly faithful and still be wrong. If the retrieved context is stale, incomplete, or drawn from the wrong source, the generator can produce a fully grounded answer that is confidently incorrect. A faithful answer to bad evidence is still a bad answer.

Common mistake: Treating a high faithfulness score as proof the answer is correct. Faithfulness only proves the answer stayed grounded. It does not prove the evidence was right, current, or sufficient.

The practical check is simple: read the answer and the retrieved chunks side by side. Mark each claim as supported, contradicted, or absent. If you see a pattern of unsupported claims, the generator is drifting from its evidence—look at how the context is assembled and how the prompt instructs the model to use it.

Relevance: Does the Answer Actually Address the Question?

An answer can be fully grounded and still useless. Relevance measures whether the answer directly addresses the user's question, not just whether it is factually supported by the context.

The classic failure mode is the faithful answer to a different question. The system retrieves relevant material, the generator produces an accurate summary of that material, and the response misses what the user actually asked.

Imagine a user asks for a comparison between two approaches. The system responds with a clean definition of each approach. Every claim is supported by the retrieved context. The answer is faithful. It is also useless, because the user asked for a comparison and got two separate descriptions.

The distinction matters because relevance failures point to a different part of the system than faithfulness failures. When an answer is unfaithful, the generator is inventing or contradicting evidence. When an answer is faithful but irrelevant, the problem is usually question understanding or prompt instruction—the model answered a different question than the one on the desk.

Note: Retrieval relevance and answer relevance are different axes. Retrieval relevance asks whether the right chunks were found. Answer relevance asks whether the generated response serves the actual query. You can have perfect retrieval relevance and terrible answer relevance.

The practical check: restate the question, then ask whether the answer would satisfy someone who asked exactly that. Watch for answers that drift to adjacent topics—the model answering a question that is related to the one asked but not the same as it.

Completeness: Did the Answer Cover What the Question Required?

Completeness is the easiest lens to miss, because a partial answer often reads as confident and complete. Completeness measures whether the answer covers all parts of the question, not just one aspect of it.

The failure mode shows up most clearly with multi-part questions. A user asks, "What are the risks and benefits of X?" The system returns a thorough explanation of the benefits. Every claim is supported. The answer directly addresses the question. It is also incomplete, because the risks were never covered.

Completeness failures also appear when a question requires synthesis across several retrieved chunks, and the generator answers from a single chunk. The evidence for a full answer was on the desk, but the model stopped at the first plausible piece and never assembled the rest.

Why is completeness so easy to miss? Because a partial answer does not look broken. It reads smoothly, sounds authoritative, and may even be correct as far as it goes. If the reader already knows part of the answer, the missing piece is even harder to spot.

The practical check: decompose the question into its required sub-answers before you read the response. If the question asks for risks and benefits, write down "risks" and "benefits" as two required pieces. Then verify each one is present in the answer.

The Three Lenses in One Table

Lens	Question it answers	Failure it catches	Symptom you would notice
Faithfulness	Is every claim supported by the retrieved context?	The generator invented or contradicted evidence	The answer asserts something not in the context, or contradicts it
Relevance	Does the answer address the question that was asked?	The generator answered a different question	The answer is accurate but does not answer what was asked
Completeness	Did the answer cover everything the question required?	The generator covered only part of the question	The answer handles one aspect but misses required pieces

None of these three lenses measures retrieval quality. They judge what the generator did with the evidence it received. If you want to know whether the retriever found the right evidence in the first place, that is a separate evaluation with its own metrics.

Diagnosing a Weak Answer: Which Lens Points Where?

When an answer looks wrong, run it through the three checks in order. Faithfulness first, because it is the most fundamental: is the answer grounded at all? Then relevance: does it answer the question? Then completeness: did it cover everything?

The order matters because each check narrows the suspect list.

If faithfulness fails, the generator is inventing or contradicting. Look at prompt grounding, context assembly, or model behavior. The evidence may have been fine, but the generator did not stay anchored to it.

If faithfulness passes but relevance fails, the problem is question understanding or prompt instruction. The model answered a different question than the one asked. This is not a retrieval problem and not a hallucination problem—it is a targeting problem.

If faithfulness and relevance pass but completeness fails, you have reached the boundary where retrieval and generation blur. The generator may have missed evidence that was retrieved, or retrieval may have failed to bring back everything the question required. This is the one case where you need to look at both sides before assigning blame.

Common mistake: Jumping straight to "swap the model" when the real fix is prompt grounding, question decomposition, or retrieval coverage. A different model will happily reproduce the same failure if the underlying contract is broken.

The decision rule is simple: name the failing lens before you change anything. If you cannot say whether the answer was unfaithful, irrelevant, or incomplete, you are guessing at the fix.

Where Answer Evaluation Stops and Retrieval Evaluation Begins

The clean split is this: retrieval metrics judge the evidence found; answer metrics judge what the generator did with that evidence. Context precision, context recall, and context relevance tell you whether the right chunks were retrieved. Faithfulness, relevance, and completeness tell you whether the generated answer served the user.

There is one overlap zone worth naming. Completeness failures often trace back to retrieval gaps—the evidence needed for a full answer was never retrieved in the first place. When completeness fails, check retrieval coverage before blaming the generator. When faithfulness fails, the generator is the prime suspect.

Answer evaluation also has hard limits. It cannot tell you whether the retrieved context was the right context, whether it was current, or whether it was complete. A system can score well on all three answer lenses and still return a wrong business answer because the source document was stale. That is a context trustworthiness problem, and no amount of answer-side scoring will catch it.

The durable mental model: retrieval decides what evidence is on the desk. The generator decides what it does with that evidence. Evaluate each contract separately, and when an answer disappoints, run it through the three lenses in order—faithfulness, relevance, completeness—and name the failing one before you change anything.

That naming habit is the foundation of every useful evaluation set you will build. Once you can say "this answer was faithful and relevant but incomplete," you know where to look. Once you can say "this answer was fluent but unfaithful," you know the generator is the problem. The diagnosis comes first. The fix follows.