RAG Query Rewriting Explained: Helping Retrieval Ask the Right Question

Your knowledge base has the answer. You know it does. You wrote the documents, chunked them carefully, embedded them, and verified the chunks look right. Yet when a user asks a reasonable question, retrieval returns noise, and the LLM confidently answers from context that misses the point.

The system isn't broken in the way you expect. The documents exist. The embeddings are fine. The retriever is doing its job. The problem is that the retriever is searching for the wrong thing—because the user's question, taken literally, is a poor search query.

This is the query rewriting problem. Before you start rewriting everything in sight, it helps to understand when rewriting actually fixes retrieval and when it's just added complexity.

Why Your RAG Misses Documents That Are Actually There

Here's the mental model that causes most early RAG frustration: we treat the user's question as if it were a good retrieval query. It usually isn't.

People ask questions conversationally. They use shorthand, pronouns, and unstated assumptions. Documents are written declaratively. They state facts, define terms, and explain concepts in complete sentences. The gap between how we ask and how documents are written is real, and embeddings don't magically bridge it.

Think about what retrieval actually does. It takes your query, converts it to a vector, and finds chunks whose vectors are closest in embedding space. The retriever isn't reading for meaning the way you do. It's comparing a representation of the query against representations of chunks. When the query leaves out the subject, the constraints, or the domain vocabulary that would point toward the right chunks, the retriever has little to work with.

So the practical question isn't "does the query sound like the documents?" It's "does the query make the intended subject, constraints, and domain language explicit enough for the retriever to locate the right chunks?"

Before you touch the query, confirm the bottleneck is actually the query. If you haven't already, inspect what retrieval returns for a few test questions. Look at the raw chunks, not the final answer. If the chunks are relevant but the answer is bad, the problem is downstream. If the chunks are irrelevant, ask why: chunking, embedding choice, or the query itself. Query rewriting only helps when the query is the weak link.

What Query Rewriting Actually Does

Query rewriting transforms the user's question into a form that better exposes the information retrieval needs. Instead of retrieve-then-read, you run a rewrite-retrieve-read flow: transform the query first, then retrieve on the transformed version.

The rewrite can fix several distinct problems:

• Conversational shorthand: "What about the API key?" becomes "How do I configure an API key for authentication?"
• Ambiguous referents: "Does that parameter have a default?" becomes "Does the max_tokens parameter have a default value?"
• Overly terse questions: "Self-attention?" becomes "What is the self-attention mechanism and how does it work in transformer models?"
• Embedded assumptions: "Why is this slow?" becomes "What factors affect inference latency in transformer models?"

The goal is to make the concepts, entities, and constraints in the query explicit enough that the retriever can find the evidence. A vague question rewritten into a precise search query gives the retriever a much better chance of landing near the right chunks.

One important boundary: rewriting changes the query itself. It doesn't add terms to the existing query, and it doesn't split the query into pieces. Those are separate techniques with separate jobs.

Query Expansion: Adding Terms to Widen the Net

Query expansion targets vocabulary mismatch. The user says one thing; the document says another. Expansion adds related terms, synonyms, or paraphrases to the original query so retrieval can match more vocabulary variants.

Say a user asks about "heart attack treatment." Your knowledge base uses the term "myocardial infarction." A direct search might miss it. Expanding the query to include both phrasings gives the retriever a wider net.

Expansion shines when your domain has jargon, acronyms, or multiple naming conventions. Users and documents often speak different dialects, and expansion builds a bridge between them.

But expansion has a real cost. Every added term can pull in irrelevant chunks that crowd the context window. The LLM has a limited desk, and every irrelevant document takes space away from the evidence it actually needs. If you expand too aggressively, you trade a precision problem for a recall problem.

Expansion backfires when added terms drift the query toward unrelated content. Adding "treatment" to a query about diagnosis might pull in documents about therapy protocols when the user wanted testing procedures. The added terms should stay tightly coupled to the user's actual intent.

The contrast with rewriting is simple: rewriting changes the query; expansion adds to it. Rewriting makes the query clearer; expansion makes it broader.

Decomposition: Splitting Multi-Part Questions

Some questions contain several distinct information requests bundled into one sentence. "How does the transformer encoder work and what are the tradeoffs of using it for text classification?" is really two questions: one about the encoder's mechanism, one about its practical tradeoffs. A single retrieval pass will likely return chunks for one thread and miss the other.

Decomposition breaks the query into independent sub-queries, each retrieving its own evidence. Each sub-query gets its own retrieval pass, and the results are combined before generation.

This solves a real problem: when a single query embeds multiple information requests, retrieval tends to return only one thread. The LLM then answers confidently about the thread it received and ignores the rest.

Decomposition costs more. Each sub-query means another retrieval call, more latency, and more context to manage. If the question is genuinely simple—"What is the default timeout?"—decomposition is overkill. A direct search handles it fine.

The decision rule: decompose when the question contains multiple independent information requests that would each need different evidence. Don't decompose single-fact questions just because they're wordy.

A Worked Example: Following the Same Question Through Each Technique

Let's make this concrete. Suppose your knowledge base is internal documentation for a deployment tool, and a user asks a follow-up question in a chat session:

"How do I set permissions for that?"

On its own, this query is nearly useless for retrieval. "That" refers to something from earlier in the conversation—say, a service account mentioned two turns ago. The retriever has no idea what "that" means, what kind of permissions are involved, or which document would address it.

Direct search on this query will return chunks about permissions in general, scattered across unrelated topics. The retriever has no subject anchor, so it grabs whatever sits nearest to generic permission language.

Rewriting resolves the referent first: "How do I set permissions for the deployment service account?" Now the retriever has a concrete entity to match against. Chunks that mention the service account and permission configuration move to the top. This is the right intervention because the core failure is ambiguity, not missing vocabulary.

Now imagine a different failure. The user asks about "rotating credentials," but your documentation consistently uses the phrase "updating access keys." The query is clear and specific—it just uses different vocabulary than your documents. Rewriting won't help much here because the wording isn't ambiguous. Expansion is the better fit: add "updating access keys" alongside "rotating credentials" so retrieval can match both dialects.

Finally, consider a multi-part question: "How do I set up the service account and what permissions does it need?" One retrieval pass will likely favor one thread. Decomposition splits this into two sub-queries—one about setup steps, one about permission requirements—each retrieving its own evidence before the results are merged.

The same conversation can trigger different failures. Name the primary bottleneck first, then choose the smallest intervention that addresses it.

Choosing the Right Technique for the Failure

Here's the framework I use when diagnosing retrieval misses. Name the failure type first, then pick the technique that targets it.

Failure pattern	What's happening	What to try
Query is clear and retrieval returns relevant chunks	Nothing is broken	Direct search. Don't add complexity
Query is conversational, ambiguous, or poorly structured	The wording doesn't expose the intended subject or constraints	Query rewriting
User and documents use different vocabulary	Vocabulary mismatch is hurting recall	Query expansion
Question contains multiple independent requests	One retrieval pass only catches one thread	Decomposition

These failure categories can overlap. A follow-up question may need conversational rewriting and expansion. A multi-part query may also contain vocabulary mismatch. When that happens, classify the primary bottleneck first, choose the smallest intervention, and add another only if evaluation shows a remaining failure.

The common mistake is stacking all techniques without measuring which one fixes the actual failure. I've seen pipelines that rewrite, expand, and decompose every query, adding latency and complexity while obscuring which transformation actually helps.

Test one transformation at a time. Run retrieval before and after, and compare the chunks. If rewriting doesn't measurably improve retrieval, drop it. If expansion adds noise, tighten it. The technique that fixes your specific failure is the only one worth keeping.

When Query Rewriting Is Not the Answer

Query rewriting has honest limits, and respecting them saves you from chasing the wrong fix.

Rewriting cannot conjure documents that were never ingested. If the knowledge base doesn't contain the answer, no query transformation will retrieve it. The best rewrite in the world returns nothing useful when the information simply isn't there.

Rewriting also can't fix chunking problems. If chunks are too large, too small, or split mid-concept, the query isn't the bottleneck. Rewriting a great query against badly chunked documents still returns badly chunked documents.

And rewriting carries its own risks. It adds an extra LLM call and latency to every request. It can introduce factual drift, subtly changing the user's intent in ways that make retrieval worse. A rewrite that over-complicates a clear query can hurt more than it helps. Research on query rewriting has found that verbose, unnecessarily complex rewrites can fail retrieval where simpler versions succeed.

My rule: rewriting must earn its place through measured improvement. Run retrieval on the original query. Run it on the rewritten query. Compare the chunks. If the rewrite doesn't retrieve better evidence, don't keep it just because it sounds more sophisticated.

The Decision Rule

When your RAG system misses documents that exist, work through this sequence:

1. Inspect what retrieval actually returns. Look at the raw chunks, not the final answer.
2. Name the failure type. Is the query ambiguous? Is there a vocabulary mismatch? Does the question contain multiple requests? Or is the problem upstream in chunking or ingestion?
3. Pick one transformation that targets the failure you identified.
4. Measure the difference. Compare retrieval results before and after. Keep the transformation only if it measurably improves the evidence.

Query rewriting is one tool in a diagnostic workflow, not a default upgrade. The goal isn't to make every query sound more polished. It's to help retrieval ask the right question—the question that exposes the subject, constraints, and domain language needed to find the evidence.

Start with one test query that currently fails. Inspect the retrieved chunks. Name the failure. Apply one transformation. Measure the difference. That single loop will teach you more about your RAG system than any rewrite technique applied blindly.