The Candidate-Recall Ceiling in RAG Reranking

You swap in a stronger reranker. The top three results look sharper, the ordering feels smarter, and the demo lands. Then you measure recall and it barely moves. The reranker got better; the answers did not.

That gap is not a tuning problem. It is a structural one, and it has a number.

Why a Better Reranker Stops Helping

The instinct behind the upgrade is reasonable: reranking is a quality stage, so a stronger model should raise quality. That model is incomplete. A reranker does not search. It reorders.

Concretely, a reranker takes a set of candidates that retrieval already produced and returns a permutation of some subset of that set. It is a closed-candidate operator: its output is built entirely from its input. If the relevant document was never retrieved, the reranker never sees it, never scores it, and never ranks it. No cross-encoder, however strong, can promote a document that is not in the pile.

Note: A document the first stage never surfaced cannot be ranked. That single sentence is the whole article in miniature.

This is why reranking is justified only when the candidate set already contains the evidence. The reranker's job is to put the right evidence on top, not to find evidence that retrieval missed. When you treat it as a quality upgrade rather than a reordering of a fixed set, you spend effort on the wrong stage and wonder why the numbers stall.

Notation: Candidate Set, Window, and Two Kinds of Recall

Before the arithmetic, name the pieces. Assume a corpus of documents and a single query with a known set of relevant documents.

• Corpus: every document the system could retrieve.
• Relevant set $R$: the documents that actually answer the query. $|R|$ is how many there are.
• Candidate set $C$: what the first stage returns, of size $K$. This is your recall@K pool.
• Reranking window $W$: the subset of $C$ the reranker actually scores. $W$ can be smaller than $C$ when a context or latency budget caps how many candidates reach the cross-encoder.

Now separate two quantities that get conflated in practice.

Candidate recall measures how much of the relevant set survived retrieval:

$\text{candidate recall} = \frac{|C \cap R|}{|R|}$

This is recall@K. It is a property of your retriever, not your reranker.

Conditional reranker recall measures how much of the retrieved relevant evidence the reranker is even allowed to see:

$\text{conditional reranker recall} = \frac{|W \cap R|}{|C \cap R|}$

This is a property of your window size, not your reranker's intelligence.

Picture a funnel: corpus → candidate set $C$ → window $W$ → final top-k. Relevant documents are marked as they fall through. By the time you reach the reranker, some are already gone, and the funnel shows exactly where.

Warning: The closed-candidate assumption is doing real work here. Systems that re-retrieve mid-generation, fuse multiple sources, or generate candidate documents operate outside this model, and the ceiling below does not bound them.

Deriving the End-to-End Recall Ceiling

End-to-end recall is the fraction of the relevant set that reaches the final output. Build it from the two quantities above.

The relevant documents that survive to the window are $|W \cap R|$. Divide by the full relevant set:

$\text{end-to-end recall} = \frac{|W \cap R|}{|R|}$

Now factor that same quantity through the two stages:

$\frac{|W \cap R|}{|R|} = \frac{|C \cap R|}{|R|} \times \frac{|W \cap R|}{|C \cap R|}$

The left factor is candidate recall. The right factor is conditional reranker recall. The middle term $|C \cap R|$ cancels — and that cancellation is the entire point.

$\text{end-to-end recall} = \text{candidate recall} \times \text{conditional reranker recall}$

Because conditional reranker recall is a fraction, it can never exceed 1. So:

$\text{end-to-end recall} \le \text{candidate recall}$

Equality holds only when the window contains every retrieved relevant document — that is, when conditional reranker recall is 1. This is the reranking candidate recall ceiling: the reranker lives under a bound that retrieval set, and it cannot lift that bound no matter how good it gets.

The ceiling binds in two ways. A candidate set that is too small ($K$ too low) drops relevant documents before reranking. A window $W$ narrower than $C$ drops retrieved relevant documents before the reranker scores them. Both are losses the reranker cannot recover.

What reranking actually does is redistribute recall across ranks. It changes where relevant evidence lands in the final list, not whether it exists in the pipeline. That is a real and valuable job. It is just not the job of raising coverage.

Worked Example: A Labeled Query with Five Relevant Documents

Numbers make this concrete. Suppose a query has $|R| = 5$ relevant documents in the corpus. The first stage returns $K = 20$ candidates, and 4 of the 5 relevant documents are inside that set.

$\text{candidate recall} = \frac{4}{5} = 0.80$

One relevant document is already unreachable. No reranker will ever see it.

Now set the reranking window to $W = 10$. Of the 4 retrieved relevant documents, 3 fall inside the window:

$\text{conditional reranker recall} = \frac{3}{4} = 0.75$

End-to-end recall follows:

$\text{end-to-end recall} = \frac{3}{5} = 0.60$

Check it against the product: $0.80 \times 0.75 = 0.60$. The relationship holds.

Now watch the ceiling. A perfect reranker over this window caps end-to-end recall at 0.60 — it can promote all 3 relevant documents inside $W$, but it cannot touch the fourth retrieved document outside the window, nor the fifth that retrieval never found. The reranker's quality is irrelevant to that number. The ceiling was decided before it ran.

Solving for the Candidate Recall You Actually Need

The relationship inverts, which is what makes it useful for planning. Start from a target instead of a guess.

Suppose you need end-to-end recall of 0.90, and you have measured conditional reranker recall at 0.75. Rearrange the product:

$\text{required candidate recall} = \frac{\text{target end-to-end recall}}{\text{conditional reranker recall}}$

Plug in the numbers:

$\text{required candidate recall} = \frac{0.90}{0.75} = 1.20$

A required candidate recall above 1.0 is impossible. That is not a rounding error — it is proof that the target is unreachable at this window size. No amount of retrieval improvement gets you there, because the window is throwing away a quarter of the retrieved relevant evidence before the reranker scores anything.

The fix path has two levers. Widen the window to raise conditional reranker recall, or improve retrieval to raise candidate recall. Try widening $W$ until conditional reranker recall reaches 0.95:

$\text{required candidate recall} = \frac{0.90}{0.95} \approx 0.947$

Now the target is reachable, but it demands candidate recall near 0.95 — a real retrieval bar, not a reranker setting.

Common mistake: Tuning the reranker when the required candidate recall already exceeds what your retriever achieves. If retrieval cannot hit the number, no reranker closes the gap.

One caveat on measurement: both quantities need a labeled evaluation set, and small samples make these ratios noisy. A handful of queries will produce candidate recall figures that swing by several points. Treat the arithmetic as a planning tool, not a precise instrument, until your fixture is large enough to trust.

Diagnosing Which Stage Is the Bottleneck

The arithmetic becomes a diagnostic once you run it in order.

1. Measure recall@K at your current $K$.
2. Measure conditional reranker recall at your chosen window.
3. Compare the product against your target.

The pattern of the two numbers tells you where to work.

When both are low, fix retrieval first. It sets the ceiling, so improving it raises the maximum that every downstream stage can reach.

Two mistakes show up again and again. The first is reading a strong NDCG or a clean top-3 as proof that coverage is fine. Ordering metrics can look stable while long-tail recall quietly degrades across query variants — the top of the list stays sharp while the tail rots. The second is widening $K$ without checking whether the recall curve has already flattened, which adds reranker cost for near-zero coverage gain.

The decision boundary is the shape of the recall@K curve. When recall is still climbing steeply at your current $K$, widen before you touch the reranker — you are leaving retrievable evidence on the table. When the curve has plateaued, more $K$ buys almost nothing, and the move is to change the retrieval method: hybrid search, domain-adapted embeddings, or better chunking.

What Breaks the Ceiling Argument

The bound is precise, so it is worth stating where it stops applying.

It assumes a closed-candidate reranker. Systems that re-retrieve during generation, fuse multiple retrieval sources, or generate candidate documents are not bounded by this relationship, because their candidate set is not fixed before ranking. The bound is also about coverage, not answer quality: a pipeline can hit its recall ceiling and still produce a poor answer for reasons entirely downstream of retrieval. And recall ceilings measured on one corpus do not transfer automatically — the numbers are properties of your retriever and your labeled set, not universal constants.

The practical next step is to build a small labeled query fixture and plot recall@K across a range of $K$. Find the knee — the point where the curve flattens. That knee tells you where to set your candidate budget and your reranking window, and it tells you whether your next hour belongs with the reranker or upstream with embeddings, hybrid search, and chunking.

Measure candidate recall before you tune the reranker. The reranker lives under a ceiling it did not build, and the levers that raise it are the unglamorous ones upstream.