RAG Reranking Explained: Choosing the Best Evidence After Retrieval

Your retriever returns ten chunks. They all look topically related to your question. None of them actually answer it. The LLM dutifully composes a response from the noise, and the result is polished, plausible, and wrong.

This is the classic failure mode that RAG reranking exists to solve. The problem is not that retrieval failed entirely. The problem is that your retriever was never designed to make the final call on relevance.

Why Your Retriever's Top Results Aren't Good Enough

First-stage retrieval in a RAG pipeline is built for speed and recall, not precision. When you run an embedding search over a large corpus, the system has to scan millions of chunks cheaply and return plausible candidates. That means it compares your query against pre-computed embeddings using a similarity metric.

Here is the catch: semantic similarity measures how close two meanings are. It does not measure whether a chunk contains the factual answer your query needs.

A chunk about "renewable energy policy in California" will sit close to a query about "California solar incentives" in embedding space. But if the chunk discusses regulatory history without mentioning the specific incentive program you asked about, it is topically related and practically useless.

You will see this symptom clearly: the retriever returns chunks that are on-topic but miss the specific evidence, so the LLM answers from noise. The retriever did its job. The job was just too coarse.

The mental model that fixes this is a two-stage pipeline. A fast, broad retriever narrows millions of chunks to tens. Then a slower, more precise relevance stage narrows those tens to the few that actually matter. That second stage is reranking.

What a Reranker Actually Does Differently

To understand what a reranker does differently, you need to see how the first stage compares text.

Embedding retrieval encodes the query and each chunk independently. The query becomes one vector. Each chunk in your corpus becomes another vector. Then the system measures the distance between them. This is called a bi-encoder architecture, and it is what makes fast search possible: every chunk is encoded once, offline, and stored for quick comparison.

A reranker works differently. Most rerankers use a cross-encoder architecture, which processes the query and each candidate chunk together in the same model. Instead of comparing two independent summaries, the model reads the full question against the full candidate passage and judges fit directly.

Here is a plain-language translation: the retriever compares summaries side by side. The reranker reads the question against each full candidate and decides whether the candidate actually answers it.

That joint processing lets the reranker weigh exact wording, negation, and fine-grained relevance signals that get lost when text is compressed into a single embedding vector. A query like "Which states do not require solar permits?" demands a different answer than "Which states require solar permits?" A bi-encoder may place both chunks nearby. A cross-encoder can catch the distinction because it sees the full query and full passage together.

Rerankers output a relevance score for each query-passage pair. That score gives you two capabilities at once: you can reorder candidates by predicted relevance, and you can filter out low-scoring chunks entirely before they reach the generator.

Retriever (fast, broad) → many candidates
        ↓
Reranker (slow, precise) → scores each query-passage pair
        ↓
Top-k high-scoring chunks → generator

Recall vs. Precision: The Tradeoff That Decides Whether You Need Reranking

Reranking is not a magic component you bolt onto every pipeline. It is a precision fix applied after a recall-oriented first stage.

Recall asks: did the retriever bring back the chunks that contain the answer at all? If the answer chunk never made it into the candidate set, no reranker can save you. You are polishing an empty deck.

Precision asks: of the chunks returned, how many are actually useful evidence versus noise? This is where reranking earns its keep.

The practical pattern that emerges from this distinction is counterintuitive at first. You retrieve a larger candidate set than you actually need, then let the reranker cut it down to a precise final set. Instead of asking your retriever for the three closest chunks, you ask for twenty or fifty, then let the reranker pick the best three.

This works because the retriever's ranking at the top of the list is less reliable than its ability to surface relevant candidates somewhere in a broader set. You trade a bit of retrieval speed for much better final precision.

So when is reranking clearly worth it? Look for these signals:

• Your corpus is large or noisy, with many near-duplicate chunks.
• Your questions involve subtle wording where negation or specific details matter.
• Your answers keep coming out wrong even though retrieval looks reasonable.
• You need high precision for a narrow set of questions.

When is it not the fix? If the answer chunk never surfaced in the candidate set, the problem is upstream in retrieval or chunking. Reranking cannot retrieve what was never found.

A Worked Example: Seeing the Reranker Change the List

Let's make this concrete. Suppose your query is "What income limit applies to California's solar incentive program?" Your retriever pulls back five candidates:

The retriever ranked the broad overview first because it shares the most vocabulary and thematic overlap with the query. But chunk 3 is the one that actually answers the question.

Now the reranker scores each query-passage pair. It reads the full query against each chunk and produces a new order:

If your final context cutoff is the top three, the reranker just changed which evidence the LLM sees. The answer-bearing chunk moved into the context window, and the merely topical overview got demoted.

Notice what the reranker did not do. It did not invent chunk 3. The chunk had to be in the retrieved candidate set first. If your retriever had only returned the top three chunks and never surfaced the income eligibility details, reranking those three would have changed nothing.

This is why the two k values matter. The retrieval k—how many candidates you pull—should be larger than the final context k—how many chunks you actually pass to the generator. Retrieve twenty or fifty. Rerank. Then keep the top three or five. If you retrieve three and rerank three, the reranker has no room to improve anything.

Where Reranking Sits in the Pipeline (and What It Cannot Fix)

Reranking sits between retrieval and context assembly. Retrieve broadly. Rerank precisely. Then pass the top few chunks to the generator.

If you have worked through the RAG pipeline stages before, you know that each stage has its own failure mode. Ingestion can miss documents. Chunking can split meaning across boundaries. Retrieval can fail to surface the right evidence. Generation can ignore or contradict the context it was given. Reranking is one stage in that chain, and it only repairs one kind of failure: poor ordering and filtering of candidates that were already retrieved.

This boundary matters because beginners often treat a reranker as a general-purpose quality upgrade. It is not. Reranking cannot fix bad chunking. It cannot fix missing documents. It cannot fix a retriever that never surfaced the right evidence in the first place.

There is another limit worth naming clearly. A reranker predicts relevance. Predicted relevance is not the same as observed usefulness. A chunk can score high on relevance and still contain outdated information, a subtle error, or a claim that does not actually support the answer. The reranker tells you which chunks are most likely to matter. It does not certify that they are correct.

Diagnosing Whether Reranking Is Your Bottleneck

Before you add a reranker, run this diagnostic sequence:

1. Is the needed evidence anywhere in your retrieved candidate pool? Look at the actual chunks your retriever returned, not the final answer. If the answer-bearing chunk never appeared, the problem is upstream. Repair your corpus, chunking, or retrieval before touching reranking.
2. Is the evidence present but buried below irrelevant chunks? If the right chunk showed up at rank 17 but never made it into the final context, reranking is likely your fix. Retrieve more candidates and let the reranker promote the right evidence.
3. Is the evidence present and highly ranked, but answers still fail? If the selected chunks are good and the LLM still produces wrong answers, the bottleneck is elsewhere. Inspect how the model uses the context or whether the chunks actually support the answer.

This sequence separates the three failure modes beginners usually conflate: evidence never retrieved, evidence retrieved but poorly ranked, and evidence retrieved and ranked but poorly used.

Common Reranking Mistakes Beginners Make

The reranking stage is simple in concept and easy to misuse in practice. Here are the mistakes I see most often.

Mistake 1: reranking the same small top-k the retriever already returned. If your retriever returns three chunks and you rerank those three, the reranker has nothing new to choose from. Retrieve more candidates first. The reranker needs a broader pool to add value.

Mistake 2: treating the reranker's score as absolute truth. Reranker scores are relative ordering signals, not calibrated probabilities. A score of 0.8 on one query does not mean the same thing as 0.8 on another. Use the scores to order and filter, not as a universal quality threshold. If you want to set a cutoff, validate it against your own queries and data.

Mistake 3: skipping the reranker because it adds latency, without measuring the tradeoff. Yes, rerankers add compute per query because each candidate must be processed jointly with the query. If you retrieve fifty candidates, that is fifty query-passage pairs to score. But if the accuracy gain matters for your use case, the latency cost may be worth it. Measure both before you decide.

Mistake 4: assuming a reranker fixes retrieval failures. If the right chunk never surfaced, reranking is polishing an empty deck. Diagnose upstream first.

Mistake 5: passing every retrieved chunk to the model instead of filtering down. The whole point of reranking is to give the generator a tight, high-signal context. If you retrieve fifty chunks and pass all fifty to the model after reranking, you have defeated the purpose. Filter to the top few.

When to Add a Reranker (and When to Skip It)

Here is the decision framework I use.

Add a reranker when answers are consistently noisy or wrong despite retrieval that surfaces the right evidence somewhere in the candidate pool. Add one when your corpus has many similar chunks that confuse the retriever. Add one when you need high precision for a narrow question set where subtle wording matters.

Skip it when your corpus is small and clean and retrieval already returns precise results. Skip it when latency and cost constraints outweigh the accuracy gain for your specific use case.

The practical middle path is to start without a reranker. Run the diagnostic sequence above. Look at whether the right evidence is surfacing in your candidate set. Only add a reranker when the evidence shows that ranking, not retrieval or chunking, is the bottleneck.

My rule is simple: retrieve broadly, rerank precisely, and only add the reranker when measured retrieval quality shows ranking is the problem you actually have. Test it against a small set of real queries. Compare the answers with and without reranking. If the accuracy gain justifies the added latency, keep it. If not, you have saved yourself a stage you did not need.